Is Your Trading Bot Profitable—or Just Lucky? How Many Trades Do You Need to Test an Edge?

A positive balance and a positive expectation are different claims

Suppose a trading bot finishes 200 trades with 110 winners. Its report shows a 55% win rate, positive profit and a profit factor above 1. Nothing in those figures is necessarily wrong. The problem begins when the report turns “this sample made money” into “this strategy has a repeatable edge.”

The first statement is accounting. The second concerns an unknown process that will produce other outcomes. To examine it, define the payoff being estimated, the unit counted as an observation and the alternatives that were tried before this record was chosen. A confidence score written by ChatGPT or Claude does not answer any of those questions.

Our general backtest sample-size guide explains why 100 trades is not a universal threshold. Here we audit one profitable record in detail, then construct two records with the same headline statistics. These are disclosed numerical examples, not TensorParadigm trading performance or observations from a live bot.

Begin with an account of every unit of profit

Use a fixed reporting unit of 1R = $10 throughout the examples. Each base-case trade makes +1R gross if it wins or loses −1R gross otherwise. Every completed trade incurs an assumed total cost of 0.06R, so a winner nets +0.94R and a loser nets −1.06R. The cost is a declared exercise assumption, not a current exchange quote. R here is a fixed dollar scale; it is neither a percentage account return nor a guaranteed loss limit.

The 110 winners contribute +103.4R after costs. The 90 losers contribute −95.4R. Net profit is +8R, or $80, and average net payoff is +0.04R per trade. Profit factor, calculated consistently from positive and negative net outcomes, is 103.4 ÷ 95.4 = 1.08385744. No account deposit or portfolio exposure schedule has been supplied, so there is no basis for converting $80 into an account return.

This record is profitable in the descriptive sense. We will not make it unprofitable by changing a statistical label. The question is whether it provides persuasive evidence that the underlying expected net payoff is positive. Our profit-factor guide covers the accounting metric; the remaining sections examine the inference.

The relevant break-even rate is 53%, not 50%

Let p be the probability of a winning outcome under this deliberately simplified fixed-payoff contract. Expected net payoff is μ = 0.94p − 1.06(1 − p) = 2p − 1.06, measured in R per trade. It equals zero at p = 0.53. Winning more often than losing is therefore not the appropriate profitability hurdle for this contract.

The hypothesis of interest is H0: μ ≤ 0 against H1: μ > 0. With these known payoffs and costs, that becomes H0: p ≤ 0.53 against H1: p > 0.53. An exact binomial upper-tail calculation at p = 0.53 gives the most adverse boundary of that composite null. This calculation assumes independent outcomes with a common winning probability. “Exact” describes that sampling model; it does not make the model an exact description of a market. [2]

mu = 0.94*p - 1.06*(1-p) = 2*p - 1.06   [R per trade]
p_break_even = 1.06 / (0.94 + 1.06) = 0.53
H0: p <= 0.53
H1: p > 0.53
p_value = sum[choose(200,k) * 0.53^k * 0.47^(200-k), k=110,...,200]

For a different fixed gross win W, gross loss magnitude L and per-trade cost c, expected net payoff is (W + L)p − L − c, and break-even p is (L + c)/(W + L). If W, L or costs vary and are estimated from the same data, their uncertainty must also be considered. Replacing them with sample averages and applying a win-rate interval alone does not account for that uncertainty.

What the 110 wins actually support

For 110 wins out of 200, the exact one-sided p-value against the 53% boundary is 0.31042765. It is the probability, under that boundary model, of obtaining at least 110 wins. It is not a 31% probability that the bot has no edge, a 69% probability that it works or a forecast of the next month’s result. The observed record does not reject the stated null at a predeclared 5% significance level.

A two-sided 95% Wilson score interval for p is approximately 48.0756% to 61.7359%. Wilson is an approximate interval, distinct from the exact tail calculation. Because this example fixes both payoff sizes and cost, transforming its endpoints through μ = 2p − 1.06 gives approximately −0.0985R to +0.1747R per trade. That range includes negative and positive expected net payoffs. [1]

The interval describes uncertainty about an underlying parameter under the model. It is not a prediction interval for the next trade or the next 200 trades, and frequentist coverage does not assign a 95% probability to the particular parameter lying inside this realized interval. Also keep the tail conventions visible: a two-sided 95% interval and a one-sided 5% test use different allocations of error.

Ignoring costs and testing against a 50% win probability would give an upper-tail p-value of 0.08948202. That answer concerns a different null. Statistical software can calculate it perfectly while answering the wrong economic question. The first task in trading strategy backtesting is to define the claim that the calculation is meant to test.

Three identical summaries can describe different evidence

Now change the structure beneath the report without changing its total profit, winning fraction or profit factor. Record A contains 200 separately labeled events; independence is an explicit assumption for its binomial analysis. Record B contains 20 independent episodes, each copied into ten perfectly linked fills. Record C has variable winner sizes, including one large winner. None of these constructed labels proves independence or stability in real data.

All three records show 200 rows, 110 positive net outcomes, +8R and a net-outcome profit factor of 1.08385744. The headline report cannot identify which record produced it. Event IDs, exposure grouping and the full payoff distribution carry information that the aggregate metrics omit.

Two hundred fills can contain only twenty independent outcomes

In Record B, each of the 20 episodes is an independent Bernoulli outcome under the assumed design. All ten fills within an episode win or lose together. There are 11 winning episodes and 9 losing episodes. Counting 110 wins in 200 rows as independent trials would count the same episode outcome ten times.

Each fill still uses the $10 reporting-risk unit. Ten fills therefore represent $100 of episode risk, not $10. A winning episode nets +9.4R in the original fill-R unit, and a losing episode nets −10.6R. The aggregate remains +8 fill-R = $80. If an episode-risk unit is defined as $100, the same $80 is +0.8 episode-R. Relabeling the unit does not create or destroy profit.

The relevant binomial count is 11 wins in 20 episodes. Its exact one-sided p-value against p = 0.53 is 0.51960841. Its two-sided 95% Wilson interval for the common episode win probability is approximately 34.2085% to 74.1802%. In the original $10 fill-R scale, the corresponding expected payoff per fill ranges from about −0.3758R to +0.4236R. The point estimate remains +0.04R per fill; the uncertainty is much wider.

Twenty is justified here by the deliberately specified perfect-copy structure and independence between episodes. It is not a rule that every ten trades count as one. Real overlapping positions require an inference method suited to their time and cross-market dependence. Lo’s analysis of Sharpe ratios likewise shows that serial dependence changes return inference; a count of trade IDs cannot simply replace the appropriate observation structure. [4]

One large winner changes the question about the mean

Record C keeps total gross winning profit at 110R: 109 × 0.75R + 28.25R = 110R. Losses and costs remain unchanged, so the win rate, net profit and profit factor match Record A. Yet the largest net winner is +28.19R. Removing that one row leaves −20.19R over the remaining 199 rows. In Record A, removing any one net winner leaves +7.06R.

This removal is a concentration diagnostic. It is not a p-value and does not establish that the large winner is illegitimate. A trend-following strategy may be designed to earn infrequent large gains. What matters is whether the sample supports the frequency and size of those gains well enough to estimate the mean. Deleting them after seeing the result can discard the strategy’s intended mechanism.

The fixed-payoff transformation μ = 2p − 1.06 is inapplicable to Record C’s variable payoffs. Even a well-estimated winning probability leaves uncertainty about the amounts won and lost. A suitable mean-payoff analysis must retain their distribution and dependence. The observed average win of 1R happens to match Record A; it does not make every future winner a known +1R payment.

A stronger result can become weak evidence after selection

Consider a separate fixed-payoff record with 120 wins out of 200. At the same cost it earns +28R, and its one-sided exact p-value against p = 0.53 is 0.02751749. For one prespecified, fixed-horizon test with valid independent observations, it would cross a 5% rejection threshold. That description changes if it was the best result selected from a large search.

For a transparent benchmark, suppose 100 candidate records are independent and every candidate truly has p = 0.53. Let q = 0.02751749 be the chance that one null candidate reaches at least 120 wins. The probability that at least one reaches that count is 1 − (1 − q)^100 = 93.8598%. This is a constructed all-null search calculation, not a measured search or a 93.86% posterior probability that an actual selected bot is false.

q = P(K >= 120 | K ~ Binomial(200, 0.53)) = 0.02751748874
P(at least one such result among 100 independent null candidates)
  = 1 - (1-q)^100
  = 0.9385983575
Bonferroni per-test level for a 0.05 family-wise bound = 0.05/100 = 0.0005

Real parameter variants often share trades and are not independent. The 93.8598% calculation should not be transplanted into such a search. A simpler Bonferroni family-wise check, for 100 declared tests with individually valid p-values, uses 0.05 / 100 = 0.0005 per test. The selected 0.02751749 does not pass it; its Bonferroni-adjusted p-value is capped at 1. This bound does not require independent tests, though it can be conservative.

Search history includes indicator thresholds, markets, filters, feature sets and prompts that were tried and abandoned. Research on the Deflated Sharpe Ratio, multiple factor testing and data snooping addresses this broader selection problem. Those methods have their own assumptions; they do not supply a universal Sharpe or t-statistic hurdle for every bot. Record the search and preserve a genuinely untouched final evaluation. [5] [6] [7]

How many trades would detect this particular edge?

To plan sample size, specify the improvement worth detecting before evaluating a future record. Here the null remains p0 = 0.53. Suppose the planning alternative is p1 = 0.55, equivalent to +0.04R expected net payoff per trade under the fixed contract. We use 0.55 as an assumed scenario, not as a known truth inferred from the observed 55% record.

Choose a one-sided 5% significance level and target power of 80%. Power is the probability that this design rejects the null when the stated alternative is true. It is not the probability that a strategy is profitable. At n = 200, an exact test must require at least 119 wins to keep its null rejection probability no greater than 5%; its attained probability is 3.7898%. At a true p of 0.55, its power is only 11.3240%. A small real edge can therefore be difficult to detect in 200 observations.

The normal-approximation planning formula below gives about 3,841.81 observations for that small improvement. For a simple fixed design, round upward to the next hundred: n = 3,900. Enumerating the exact binomial tail gives a rejection count of 2,119 wins and power of 80.3197% at p = 0.55. The count is an illustrative design under explicit assumptions, not an absolute minimum or a universal requirement for trading bots. [3]

n_approx = [z_(1-alpha)*sqrt(p0*(1-p0))
            + z_(power)*sqrt(p1*(1-p1))]^2 / (p1-p0)^2
p0 = 0.53
alpha = 0.05   [one-sided]
target_power = 0.80
n_fixed = 100 * ceil(n_approx / 100)
k_critical = smallest integer k with P_p0(K >= k) <= 0.05
actual_power = P_p1(K >= k_critical)

Larger assumed improvements need less data. The table applies the same rule—round the normal approximation upward to the next hundred—then checks the exact integer rejection count and power. The three attained null rejection probabilities are 4.9174%, 4.6435% and 4.8929%, respectively. Discreteness is why exact designs need their actual operating characteristics checked rather than treating a rounded approximation as a guarantee.

These calculations address a fixed, independent, common-probability process with known payoff sizes and cost. They do not tell a researcher to collect trades until the observed win rate becomes significant. Nor does 3,900 observations in one market environment establish persistence in another. A tiny assumed effect can require an evaluation too long for the stability of the strategy’s premise; that is a research constraint, not a reason to lower the evidence standard.

Checking every week changes the experiment

A fixed-horizon test concerns the result at its declared endpoint. If a researcher inspects it every week and stops at the first favorable p-value, the reported result comes from a different selection rule. Repeatedly trying to cross an ordinary fixed-time threshold need not preserve its nominal error rate. Deciding after the fact which stopping date counts adds another search dimension.

A predefined endpoint is one solution. A suitable sequential design is another. Howard and colleagues develop confidence sequences with simultaneous coverage over time under their stated conditions, supporting valid monitoring within that framework. Such methods are not assumption-free substitutes for an arbitrary heavy-tailed, changing return process. This article does not implement or report a sequential trading test. [8]

An operational loss limit can still stop trading at any time. Protecting capital does not require waiting for 3,900 trades or for a statistically significant loss. Keep that safety decision separate from a claim of statistical success. Retain the stopped record, explain the intervention and identify how it changes the evaluation; do not quietly erase a failed run and restart the evidence count.

Turn the next evaluation into evidence that is harder to select

Before the next evaluation, write a short decision record. Freeze the strategy version, eligible instruments, information timing, cost treatment, observation unit, primary outcome and statistical endpoint. Keep every eligible decision and distinguish submitted orders, actual fills and paper assumptions. Our AI trading-decision investigation explains why those records differ.

For the narrow binary example, the table shows what a fixed design would have to declare. A real strategy with variable sizes, overlapping positions or changing costs needs a different justified analysis. Do not borrow the 2,119-win rule simply because the trade count happens to reach 3,900.

Trading strategy optimization must stay on the development side of the boundary. Choosing a new prompt or filter after inspecting the final period makes that period part of selection. Walk-forward analysis can evaluate an explicitly specified updating procedure, but its chronology must preserve which outcomes were available before each update. Our algorithm-building study covers those mechanics.

Evidence of positive net mean is also narrower than evidence of alpha, adequate capacity or an acceptable loss path. Compare against relevant alternatives with consistent capital and timing, and evaluate drawdown and execution separately. The record may reflect market exposure rather than a unique forecasting contribution. A statistically detectable effect can also be too small to justify fixed operating costs; the capital and break-even guide addresses that separate economic question.

What AI and simulation can help calculate—and what they cannot add

A ChatGPT trading bot and a Claude trading bot face the same evidence problem. The language model used to write the strategy does not make its outcomes independent or its profit statistically persuasive. Codex or Claude Code can help reconcile a trade ledger, calculate a cost-adjusted binomial tail and check a proposed evaluation design. Ask for the null, observation unit, payoff assumptions, tail convention and search history alongside the numerical answer.

The useful request is specific: explain whether the reported method applies to these outcomes, identify every estimated input and show the calculation. A report that says “95% confidence the bot will profit” without identifying the event and model has skipped the hard part. Verify arithmetic against the record and the relevant statistical method; polished prose is not another observation.

A Monte Carlo trading simulation can explore consequences conditional on an input model. It does not increase the number of market events observed. With fixed positions and additive payoffs, every pure reordering of Record A’s same 200 outcomes still ends at +8R. The path can change, but shuffling alone cannot create evidence for a positive mean. If positions depend on equity, cash, overlapping exposure or stopping rules, shuffling trades also fails to rerun those decisions.

Resampling with replacement can produce a distribution of totals, but those outcomes are generated from the selected sample and the resampling assumptions. They do not observe an absent crash or independently confirm the frequency of Record C’s large winner. Use simulation to inspect sensitivity and risk under stated assumptions, while keeping prospective out-of-sample testing as a separate source of evidence.

What we can—and cannot—say about the profitable record

The base record earned $80 after its assumed costs. Under the declared fixed-payoff, independent-outcome model, its 110 wins do not provide strong evidence against zero net expectancy: the exact one-sided p-value is 0.3104, and the two-sided mean-payoff interval includes negative values. This does not establish that the strategy has no edge. It establishes that this record does not resolve the question well under that model.

The two alternate ledgers show why more summary metrics are not enough. Linked fills leave the account profit unchanged while reducing independent information. Concentrated winners leave win rate and profit factor unchanged while changing the problem of estimating the mean. Selection can then make a stronger-looking result unconvincing even before either issue is addressed.

How many trades are enough therefore begins with “enough to detect which effect, under which design?” For our specified small alternative, the chosen fixed design uses 3,900 independent outcomes for about 80% power. For a real trading system, first justify the observation unit, after-cost outcome, search boundary and monitoring rule. A profitable trading algorithm is an economic claim; its historical profit is the beginning of the investigation, not its conclusion.