What Backtesting Actually Tells You (And What It Doesn't)

We've built and backtested dozens of trading systems over the years. Some of them looked spectacular on paper — smooth equity curves, high Sharpe ratios, minimal drawdowns. We deployed them with confidence. And then watched many of them underperform or fail in live markets. This experience taught us more about backtesting than any textbook could.

The problem wasn't that backtesting is useless. It's that backtesting is seductively misleading if you don't understand its limitations. A backtest doesn't tell you what a system will do. It tells you what a system would have done under a specific set of conditions that will never repeat exactly. The gap between those two things is where traders lose money.

What Backtesting Is Good For

Before listing the traps, let's acknowledge what backtesting genuinely accomplishes when done properly. It can validate or invalidate an idea. If a strategy concept doesn't work on historical data — even under ideal conditions — it's almost certainly not going to work in live markets. A backtest that shows a negative expectancy on 10 years of data is a clear "no" signal. That's valuable. It prevents you from wasting capital on flawed ideas.

Backtesting also reveals the behavioral characteristics of a system: its typical win rate, average win vs. average loss, maximum drawdown, longest losing streak, and recovery time. These characteristics help you decide whether you can psychologically tolerate trading the system. A system with a 35% win rate and 3:1 reward-to-risk is mathematically profitable — but can you handle losing 13 out of 20 trades in a row? The backtest tells you that's a realistic possibility, so you can make an informed decision before real money is at stake.

Finally, backtesting helps compare systems against each other. If you have three strategy concepts, testing all three on the same data lets you see which one has better risk-adjusted characteristics. Not which one will perform best in the future — but which one has the most robust historical profile.

The Five Traps That Fool Everyone

Trap 1: Overfitting (The Most Common Killer)

We covered this in our automation article, but it deserves reinforcement because it's the single most common backtesting error. Overfitting happens when you optimize too many parameters on historical data. The system isn't capturing a real market inefficiency — it's memorizing the specific price sequences in your test period.

The red flag: every time you add a parameter or filter, the backtest results improve. In reality, every parameter you add makes the system more fragile. The rule of thumb we follow: if a system needs more than 3–4 parameters to work, it's probably overfit. The simpler the rules, the more likely the edge is real.

Trap 2: Survivorship Bias

Most stock databases only include stocks that currently exist. Companies that went bankrupt, got delisted, or were acquired and absorbed have been removed from the data. If you backtest a strategy on today's stock universe and project backward, you're testing only on stocks that survived — which introduces a systematic upward bias.

A momentum strategy tested on today's NIFTY 500 will look better in the backtest than it would have performed in real time, because the stocks that had terrible momentum and eventually failed are no longer in the universe. Your backtest never had the chance to buy them and lose money — but in real time, you would have.

The fix: use a survivorship-bias-free database, or at minimum, be aware that your results are likely 2–5% per year more optimistic than reality.

Trap 3: Look-Ahead Bias

This is subtler. Look-ahead bias occurs when your backtest uses information that wouldn't have been available at the time the trade was taken. For example, using the day's closing price to make a decision that would have needed to be made before the close. Or screening for "stocks that broke out this week" and entering at Monday's open — when in reality, you wouldn't have known about the breakout until it happened.

Even small look-ahead biases can dramatically inflate backtest results. The fix is simple in principle but tedious in practice: at every decision point in your backtest, ask "would I have had this information at this exact moment?" If the answer is no, the backtest is contaminated.

Trap 4: Ignoring Transaction Costs and Slippage

A backtest that assumes you buy at the exact price you want, sell at the exact price you want, and pay zero commissions is fantasy. In the real world: you get slippage on entries (especially on breakout day when the stock gaps up), you get slippage on exits (especially when your stop is triggered in a fast move), and you pay brokerage, STT, exchange charges, and GST on every transaction.

For a swing trading system doing 50–100 trades per year, assuming 0.5% round-trip cost (entry slippage + exit slippage + fees) is conservative. On a system that makes 15% per year gross, that's 3–5% eaten by friction, bringing your net return to 10–12%. Many backtests that show 25% annual returns become 15% after realistic cost assumptions — still good, but very different from the headline number.

Trap 5: Data-Mining Bias

If you test 100 different strategy variations and pick the one that performs best, you haven't found an edge — you've found the variation that happened to fit the data most closely. This is data-mining bias, and it's distinct from overfitting (though related). Even with a simple, two-parameter system, if you test it across 50 different parameter combinations, the "best" one is likely benefiting from randomness rather than genuine alpha.

The fix: have a hypothesis before you test. Decide what you're looking for and why before you look at results. If you believe volatility contraction precedes breakouts (because of the supply-demand logic), test that specific hypothesis. Don't test 50 variations and pick the winner.

What a Good Backtest Looks Like

A trustworthy backtest doesn't show the highest returns. It shows consistent behavior across different time periods and market conditions. If a system works in bull markets, bear markets, and sideways markets — even if it's not spectacular in any single one — that's a far more reliable result than a system that crushes it in one period and falls apart in another.

Specifically, you want to see: positive expectancy (average win × win rate minus average loss × loss rate is positive), a maximum drawdown you can psychologically survive, similar performance characteristics in the in-sample and out-of-sample periods, and no dependency on a single outlier trade for the overall profitability.

The last point is critical. If you remove the best three trades from a 100-trade backtest and the system goes from profitable to unprofitable, the "edge" is an illusion. It's a mediocre system that got lucky three times. A robust system remains profitable even when you remove the best trades — because the edge is distributed across many trades, not concentrated in a few.

Our Approach to Validation

At Trabot, we follow a strict protocol. We form a hypothesis based on market logic (not data patterns). We define the rules before looking at results. We test on 60% of the data, then validate on the remaining 40% without any changes to the rules. If performance degrades significantly in the out-of-sample period, the system doesn't get deployed — regardless of how good the in-sample results were.

Even after this validation, we paper trade the system for a minimum period before allocating real capital. And when we do allocate, we start with a fraction of the intended position size to confirm that live execution matches backtest assumptions.

This is slow. It's conservative. It means we reject far more systems than we keep. But the ones that survive this process have a meaningfully higher probability of working with real capital — because they've been stress-tested for exactly the biases that make most backtests unreliable.