Backtesting a trading strategy refers to testing the strategy with historical data and observe their metrics, results and performance. Commonly, backtesting has some pitfalls that should be considered by any trader before they put a strategy in production. 

Data Scientist and developers should take into account issues such as Optimization Bias, Survivorship Bias and Market Impact when backtesting their trading strategies. 

Optimization Bias

Optimization Bias takes place when a trading strategy was built based on a large number of parameters. 

The excess of parameters can lead the strategy to outperform the market with sample data, but with live market data or out of sample data, the strategy might have poor performance. Therefore, a good practice is to keep the strategy simple with fewer parameters. 

Survivorship Bias

Survivorship Bias means that the dataset used for backtesting doesn’t contain the full list of stocks at every point in time, but only consider those stocks that have “survived”.

If we use a dataset that only considers stocks that have overcome drawdowns periods, and not those stocks that were delisted, or stocks that got in bankruptcy, we are introducing Survivorship Bias and the strategy returns can increase artificially. 

Market Impact

Market Impact could be important for large transactions that are executed to rebalance portfolios. This can cause price movements that are reflected by the difference between the transaction price and what the market price would have been in the absence of the transaction.

On the other hand, the market impact could increase the bid ask spread. More illiquid assets have large spreads (difference between the bid and ask price). Depending on the supply and demand dynamics of the stock and the exchange that is used to trade, the spread can vary and increase transaction costs. Market impact can be addressed by taking a conservative perspective about trade prices while backtesting.

When backtesting and sourcing trading ideas, it is a good practice to consider other insights beyond the strategy returns. These insights are the leverage, volatility required, capital requirements, and risk metrics such as maximum drawdown tolerance and Sharpe ratio.