Clean Data, Biased Output – Backtesting vs Live Trading Results

Our backtest results and live trading performance may differ greatly (and usually in a bad way).

Our backtests may tell us that there is easy money on the table, but sometimes, they are lying. There are times where our data are clean, but they do not tell us the full story.  We need to dig deeper to understand what is happening behind the scenes in the financial markets.

Let’s categorise the reasons for these biased backtests into 2 groups: 1) General Biases and 2) Specific Biases (and by specific, we mean specific to either an asset class or derivative). In this article, we focus on General Biases.

General Biases

Here are some factors that bias our backtests.

Holding Costs/Cost of Carry

Holding costs are fees paid when we hold a trade. The longer the open position is held, the higher the costs. These may not apply to all products. They generally apply to forex, CFDs and certain commodities.

Fig 1: Holding costs on a S&P500 CFD.

Fig 1: Holding costs on a S&P500 CFD.

Survivorship Bias

Official Definition of Survivorship Bias[1]:

Survivorship bias, or survival bias, is the logical error of concentrating on the people or things that “survived” some process and inadvertently overlooking those that did not because of their lack of visibility.

In simpler words:

Survivorship bias refers to the fact that people overlook entities/processes that failed because they only see successful entities/processes.

Let’s look at how survivorship bias can distort our backtest analysis due to selection problems.

The selection Problem

Imagine that we are selecting a bunch of stocks to trade. We create a list of criteria for this selection. We filter the universe of stocks listed in the US based on these criteria and choose some stocks.

And with that, survivorship bias just got to us. This universe of stocks only includes stocks that survive. There may be stocks that are delisted but fit our criteria. We need to consider those stocks as well to give us an idea of how sound our strategy is.

Black Swan Price Reconciliation

I like this one. It is stealthy to the point that new traders do not see it coming.

Fig 2: An innocent-looking black swan

Fig 2: An innocent-looking black swan

Hypothetical Example:

Sometimes, the market goes crazy. EURUSD is trading at 1.1300. A black swan event occurs and EURUSD spikes up 2000 pips. You long EURUSD 1000 pips into the 2000 pips move. You bet BIG. You are long EURUSD at 1.2300. It is now trading at 1.3300. You close the trade.

You’re laughing all the way to the bank. Then bad news hits you. You receive an email saying that “In view of this unexpected event, all trades will be cleared at 1.1800 price”. Your 2000 pips profit becomes a 500 pips lost. Your account gets wiped out. “THE GAME IS RIGGED!” you scream in the middle of the streets (on the way to the bank). And you are partially right.

True Stories:

Transaction Costs/Commissions

There are 3 types of transaction costs: 1) Fixed, 2) Variable and 3) Mix of 1) and 2).

Note that fixed transaction costs refer to costs that charge a fixed fee no matter how large your order is. Most transaction costs are either variable or mixed.

Fig 3: An example of mixed transaction costs.

Fig 3: An example of mixed transaction costs.

Variable costs are easy to model (we’ve done this before in an earlier chapter). However, accounting for fixed transaction costs (and by association, mixed transaction costs) in our backtests is slightly trickier. For fixed transaction costs, the smaller the trade size, the higher these costs are in percentage terms.

There are 2 implications. Firstly, this means that it is more difficult for traders with small account size to trade this product (not really relevant to this chapter but worth mentioning). Secondly, it is difficult to account for these costs in our MT4 backtests (this may not be an issue in other backtesters).

Things get more complicated when the transaction costs are mixed (and complicated) – they may have a tiered structure, minimum or maximum. See fig 2.

To account for fixed and mixed transaction costs, we need to modify our MT4 (if you are using other software, you may have other options to account for these costs) backtest results. Depending on whether we over or under-compensate for these transaction costs (this depends on the spread you choose and your account size), we either need to subtract or add P&L from our results.

Spreads

There are brokers that offer fixed and variable spreads. Variable spreads lead to some biased backtests. Fixed spreads do not.

The use of the word “fixed” here is misleading because in fact, both of fixed and variable spreads are considered variables costs. As the amount you trade goes up, the costs of the spread go up proportionally.

Fixed spreads charge X amount of pips per trade. Thus, the more you trade, the higher the dollar costs go (eg. 3 pips on 1 lot = $30. 3 pips on 10 lots = $300). Don’t confuse this with fixed transaction costs, where you pay a fixed dollar amount no matter how large your order is.

Variable spreads indicate that the broker will not always offer X pips per trade. It will change depending on the market conditions. The spreads usually go up in times of low liquidity or high volatility.

Variable spreads may lead to misleading backtest results as we do not know the exact spread we will get for each trade. Thus, the next best solution is to estimate these spreads – for more info, see lecture “Modelling Transaction Costs: Commissions, Spreads and Slippages”.

Slippage and Requotes

In backtesting, you will almost always get filled at the intended price (even if the market didn’t trade at that price). However, during live trading, you may not  get the price you want. You might face slippages and requotes. Let’s recall some definitions:

Slippage: Difference between the intended price and actual price of the order.

Requote (Manual Trading): Describes a situation when your order did not execute at the intended price and the broker asks if you want to execute at another (usually worse) price.

Requote (Robot/EA Trading): Describes a situation when your order did not execute at the intended price and the MT4 returns OrderSend Error 138 (Requote).

Here are some methods to minimise slippages and requotes.

  • Avoid scalping (low take profit and stop loss values, low holding period)
  • Avoid fast moving market conditions
  • Avoid periods with low liquidity
  • Avoid shady brokers (you don’t say!)
    • Always do your due diligence on brokers. Check the brokers’ background and reviews.

Inaccurate Price Simulation

Not all backtesters replicate the exact historical price movement. Some simulate the price movements. Always understand how your backtester works!

See example on MT4: https://www.mql5.com/en/articles/75

Change in contract specifications[2]

An exchange or broker may change the contract specifications of their products. For instance, they may increase margin requirements, change the settlement specifications or contract size of their products. These may lead to jumps in market prices.

The main takeaways here is – in such cases, do not take a price change at face value. Your P&L may not change proportionally to a price change. For example, increasing the margin requirement for silver may cause silver prices to fall. In your backtest, your short silver position may look like it is doing well. However, if you had traded that move in real-life, you may get a margin call and be forced to close the position.

Links to some real-life examples:

Ending Note

When the story looks too good to be true, it usually is. It is important that you understand the Market Microstructure (aka what is exactly happening behind the scenes) of what you are trading. For those who are keen to learn about Specific Bias, go Google search for “Market Microstructure of <Insert asset class or derivative>” or “How does <Insert asset class or derivative> work”. Alternatively, go check out our course!!!


AlgoTrading101 is an Investopedia-featured algorithmic trading course that doesn’t suck. Learn more about us at AlgoTrading101.

Keep up to date

 

 

[1] https://en.wikipedia.org/wiki/Survivorship_bias

[2] http://www.futuresmag.com/2011/01/31/contract-specs-know-what-you-are-trading

Lucas Liew

This dude runs AlgoTrading101.com, an algorithmic trading academy with over 13,000 students. Click on the "Author" link above to learn more about him.