More Blog Posts Coming Up!

Hey guys,

I guess you have noticed the lack of updates on this blog. We have been spending most of our time adding content to our course (https://algotrading101.com/) and have thus neglected this blog (unfortunately). =(

But no worries, we will be adding more content in the coming months! So look out for that.

Cheers,
Lucas

p.s. Stay in touch with our blog updates (below) if you want to be notified when the next post is out!


AlgoTrading101 is an Investopedia-featured algorithmic trading course that doesn’t suck. Learn more about us at AlgoTrading101.

Keep up to date

Clean Data, Biased Output – Backtesting vs Live Trading Results

Our backtest results and live trading performance may differ greatly (and usually in a bad way).

Our backtests may tell us that there is easy money on the table, but sometimes, they are lying. There are times where our data are clean, but they do not tell us the full story.  We need to dig deeper to understand what is happening behind the scenes in the financial markets.

Let’s categorise the reasons for these biased backtests into 2 groups: 1) General Biases and 2) Specific Biases (and by specific, we mean specific to either an asset class or derivative). In this article, we focus on General Biases.

General Biases

Here are some factors that bias our backtests.

Holding Costs/Cost of Carry

Holding costs are fees paid when we hold a trade. The longer the open position is held, the higher the costs. These may not apply to all products. They generally apply to forex, CFDs and certain commodities.

Fig 1: Holding costs on a S&P500 CFD.

Fig 1: Holding costs on a S&P500 CFD.

Survivorship Bias

Official Definition of Survivorship Bias[1]:

Survivorship bias, or survival bias, is the logical error of concentrating on the people or things that “survived” some process and inadvertently overlooking those that did not because of their lack of visibility.

In simpler words:

Survivorship bias refers to the fact that people overlook entities/processes that failed because they only see successful entities/processes.

Let’s look at how survivorship bias can distort our backtest analysis due to selection problems.

The selection Problem

Imagine that we are selecting a bunch of stocks to trade. We create a list of criteria for this selection. We filter the universe of stocks listed in the US based on these criteria and choose some stocks.

And with that, survivorship bias just got to us. This universe of stocks only includes stocks that survive. There may be stocks that are delisted but fit our criteria. We need to consider those stocks as well to give us an idea of how sound our strategy is.

Black Swan Price Reconciliation

I like this one. It is stealthy to the point that new traders do not see it coming.

Fig 2: An innocent-looking black swan

Fig 2: An innocent-looking black swan

Hypothetical Example:

Sometimes, the market goes crazy. EURUSD is trading at 1.1300. A black swan event occurs and EURUSD spikes up 2000 pips. You long EURUSD 1000 pips into the 2000 pips move. You bet BIG. You are long EURUSD at 1.2300. It is now trading at 1.3300. You close the trade.

You’re laughing all the way to the bank. Then bad news hits you. You receive an email saying that “In view of this unexpected event, all trades will be cleared at 1.1800 price”. Your 2000 pips profit becomes a 500 pips lost. Your account gets wiped out. “THE GAME IS RIGGED!” you scream in the middle of the streets (on the way to the bank). And you are partially right.

True Stories:

Transaction Costs/Commissions

There are 3 types of transaction costs: 1) Fixed, 2) Variable and 3) Mix of 1) and 2).

Note that fixed transaction costs refer to costs that charge a fixed fee no matter how large your order is. Most transaction costs are either variable or mixed.

Fig 3: An example of mixed transaction costs.

Fig 3: An example of mixed transaction costs.

Variable costs are easy to model (we’ve done this before in an earlier chapter). However, accounting for fixed transaction costs (and by association, mixed transaction costs) in our backtests is slightly trickier. For fixed transaction costs, the smaller the trade size, the higher these costs are in percentage terms.

There are 2 implications. Firstly, this means that it is more difficult for traders with small account size to trade this product (not really relevant to this chapter but worth mentioning). Secondly, it is difficult to account for these costs in our MT4 backtests (this may not be an issue in other backtesters).

Things get more complicated when the transaction costs are mixed (and complicated) – they may have a tiered structure, minimum or maximum. See fig 2.

To account for fixed and mixed transaction costs, we need to modify our MT4 (if you are using other software, you may have other options to account for these costs) backtest results. Depending on whether we over or under-compensate for these transaction costs (this depends on the spread you choose and your account size), we either need to subtract or add P&L from our results.

Spreads

There are brokers that offer fixed and variable spreads. Variable spreads lead to some biased backtests. Fixed spreads do not.

The use of the word “fixed” here is misleading because in fact, both of fixed and variable spreads are considered variables costs. As the amount you trade goes up, the costs of the spread go up proportionally.

Fixed spreads charge X amount of pips per trade. Thus, the more you trade, the higher the dollar costs go (eg. 3 pips on 1 lot = $30. 3 pips on 10 lots = $300). Don’t confuse this with fixed transaction costs, where you pay a fixed dollar amount no matter how large your order is.

Variable spreads indicate that the broker will not always offer X pips per trade. It will change depending on the market conditions. The spreads usually go up in times of low liquidity or high volatility.

Variable spreads may lead to misleading backtest results as we do not know the exact spread we will get for each trade. Thus, the next best solution is to estimate these spreads – for more info, see lecture “Modelling Transaction Costs: Commissions, Spreads and Slippages”.

Slippage and Requotes

In backtesting, you will almost always get filled at the intended price (even if the market didn’t trade at that price). However, during live trading, you may not  get the price you want. You might face slippages and requotes. Let’s recall some definitions:

Slippage: Difference between the intended price and actual price of the order.

Requote (Manual Trading): Describes a situation when your order did not execute at the intended price and the broker asks if you want to execute at another (usually worse) price.

Requote (Robot/EA Trading): Describes a situation when your order did not execute at the intended price and the MT4 returns OrderSend Error 138 (Requote).

Here are some methods to minimise slippages and requotes.

  • Avoid scalping (low take profit and stop loss values, low holding period)
  • Avoid fast moving market conditions
  • Avoid periods with low liquidity
  • Avoid shady brokers (you don’t say!)
    • Always do your due diligence on brokers. Check the brokers’ background and reviews.

Inaccurate Price Simulation

Not all backtesters replicate the exact historical price movement. Some simulate the price movements. Always understand how your backtester works!

See example on MT4: https://www.mql5.com/en/articles/75

Change in contract specifications[2]

An exchange or broker may change the contract specifications of their products. For instance, they may increase margin requirements, change the settlement specifications or contract size of their products. These may lead to jumps in market prices.

The main takeaways here is – in such cases, do not take a price change at face value. Your P&L may not change proportionally to a price change. For example, increasing the margin requirement for silver may cause silver prices to fall. In your backtest, your short silver position may look like it is doing well. However, if you had traded that move in real-life, you may get a margin call and be forced to close the position.

Links to some real-life examples:

Ending Note

When the story looks too good to be true, it usually is. It is important that you understand the Market Microstructure (aka what is exactly happening behind the scenes) of what you are trading. For those who are keen to learn about Specific Bias, go Google search for “Market Microstructure of <Insert asset class or derivative>” or “How does <Insert asset class or derivative> work”. Alternatively, go check out our course!!!


AlgoTrading101 is an Investopedia-featured algorithmic trading course that doesn’t suck. Learn more about us at AlgoTrading101.

Keep up to date

 

 

[1] https://en.wikipedia.org/wiki/Survivorship_bias

[2] http://www.futuresmag.com/2011/01/31/contract-specs-know-what-you-are-trading

Retail Traders vs Institutional Traders – Can we outwit the Hedge Funds?

Short answer: It is difficult but definitely not impossible to outperform hedge funds and asset management firms over the long run.

Long answer: There isn’t a direct answer to this.

Hedge funds and big trading firms have billions in capital, teams of experienced and highly qualified portfolio managers, traders and analysts, cutting edge hardware and software infrastructure and real-time access to material market information.

Retail traders have a rusty old laptop, a free trading software and questionable Wi-Fi.

Fortunately, the good news is institutional traders and retail traders have different edges in the market. Retail traders can succeed if they know how to play their game right. To understand the difference between us and them, let’s look at the advantages and disadvantages retail traders have compared to the big boys.

David-vs-Goliath

David vs Goliath!

Advantages

  1. Small Capacity

Hedge funds have large capital base and this limits the markets they can access. As retail traders, we can fly under the radar and tackle markets that doesn’t have enough liquidity to absorb the big
boys.

  1. Lack of Investment Mandate

Some hedge funds and most asset management firms have investment mandates they have to follow.
A Long-only APAC Equity Fund has to absorb the beta[1] risk of Asian equities (unless they hold 100% cash which is unlikely) even if they are bearish on Asian Equities. Retail traders on the other hand can adapt their play to the varying market conditions.

  1. Low Execution Risk

Imagine if an asset management fund wants to buy a significant stake into Company Banana. This
will definitely move the market. Other players may notice and join the trade. This pushes the price
higher and raises the fund’s entry price.

Retails traders rarely move the market. Rarely do people care what they do – maybe except for this
guy: http://www.bloomberg.com/news/2014-09-25/mystery-man-moving-japan-made-more-than-1-million-trades.html

  1. Funds experience Capital Withdrawals

Capital withdrawals tend to be disruptive to the investment strategy. Withdrawals may cause the fund
manager to liquidate his holdings to raise cash. This is especially disruptive if the asset liquidated is
illiquid since cost of liquidation is high (executing at bad prices and paying up bid-ask spreads etc).

  1. Disclosure

Large funds that are regulated have to disclose some information on their holdings. This makes it more difficult for them to outmaneuver the market.

Disadvantages

  1. Lower Fees and Spreads

Big funds have better bargaining power and can negotiate lower execution cost. Retail traders pay high commission and spreads. In most cases, retail traders are taking prices as opposed to making prices[2] (unless they trade using depth-of-market/2nd level systems). As a price taker, retail traders pay higher spreads.

  1. Best Services and Wide Range of Products

Big funds have access to prime brokerage, other support services and a wide range of financial products. These support services spend countless hours researching and executing the best deals for the funds. Retail traders do not have this luxury.

  1. Fast Information

Information is king. Funds have access to important information quicker than the general public. This
gives them an edge.

  1. Top Technology

Big funds invest top dollar into better infrastructure. These infrastructure aids their trading in terms
of research/backtesting, execution and risk management. This allows them to engage in complex trades that the retail traders cannot access.

  1. Leverage

Funds have better credit rating than the average retail trader. Hence, they are able to get better
leverage and terms. This allows them to weather tough times and increase returns with a smaller
base capital.

Moving Forward

Once we understand the different circumstances between the big boys and us, we should realise that the real question here is not “How to we outwit the big funds?”. It is “How do we find market inefficiencies that are untouched by the funds”.

Google will not touch a $20,000 per month revenue opportunity – it is too small. However, this is big enough for a one-man tech company. Similarly, we should target these pockets of alpha[3] in the market, until we grow big enough to play in the same playground as the big boys.


AlgoTrading101 is an Investopedia-featured algorithmic trading course that doesn’t suck. Learn more about us at AlgoTrading101.

Keep up to date


[1] https://en.wikipedia.org/wiki/Beta_(finance)

[2] https://en.wikipedia.org/wiki/Market_maker

[3] https://en.wikipedia.org/wiki/Alpha_(finance)

Learn Algorithmic Trading: 6 Key Components

The ancient city of Carthage stood against the Roman Empire for over a hundred years.

“Victorious warriors win first and then go to war, while defeated warriors go to war first and then seek to win.” – Sun Tze.

Finance, Mathematics and Programming.

That is what I used to tell my students when they ask what is required for building algorithmic trading systems. However, those 3 components are vague, and they do not add value to a beginner who wants to learn how to get started in algorithmic trading.

It is more useful to divide the knowledge for building algorithmic trading systems into 6 key components. You need to be acquainted with all of them in order to build effective trading systems.

Some of the terms used may be slightly technical, but you should be able to understand them by Googling.

Note: Some of these do not apply if you want to do High-Frequency Trading.

1. Market Theories

What is it?

Market theories refer to the way the markets work. This entails understanding market inefficiencies, market participants, relationships between assets/products/news/factors and price behaviour.

Why do we need to know it?

Trading ideas stem from market inefficiencies. You will need to know how to evaluate market inefficiencies that give you a trading edge versus those that don’t. You also need to learn how to take advantage of these market inefficiencies when they occur.

Keywords to Google

Market inefficiency, price/asset/market relationships, market participants, market microstructure, macroeconomics, market fundamentals, hard arbitrage, soft arbitrage, structured products, order limit book, depth of market, retail traders vs hedge funds/institutional investors, financial exchanges, exchange traded products, over-the-counter, price expectations vs reality, price action and price behaviour.

Medieval Warfare Analogy

This is akin to understanding the battlefield. Knowing what your opponent is doing in this particular battle considering the relevant factors (both your armies, terrain, goals and characteristics of each general).

2. Robot Design

What is it?

Design of an effective portfolio of trading strategies. This entails understanding how automated trading systems work individually and together.

On an individual level, an algorithmic trading strategy consists of 3 core components: 1) Entries, 2) Exits and 3) Position sizing. You’ll need to design these 3 components in relation to the market inefficiency you are capturing (and no, this is not a straightforward process).

On a portfolio level (aka multiple strategies running together), you’ll need to know how to manage a group of algorithmic trading strategies at the same time. Strategies can be complementary or conflicting – this may lead to unplanned increases in risk exposure or unwanted hedging. Capital allocation is important too – do you split capital equally during regular intervals or reward the winners with more capital?

Why do we need to know it?

Once we find market inefficiencies, we need to find the best way to exploit them. That will require you to have knowledge of how to design trading robots.

You don’t need to know advanced maths (although it will help if you aim to build more complex strategies). Good critical thinking skills and a decent grasp on statistics will take you very far.

Design involves testing for market inefficiencies (does the market inefficiency exist?) and building effective trading strategies (can I find a way to take advantage of them?). The former entails market (is it correlation or causation – or does it not matter?), statistical and infrastructural analysis. The latter entails idea generation, backtesting (testing expectancy and robustness) and optimisation (maximising performance with minimal curve fitting).

Keywords to Google

Entries, exits, position sizing, money management, idea generation, backtesting, robustness, strategies that adapts to the market, optimisation, curve fitting, walk-forward optimisation, math/statistics for finance, correlation, cointegration, execution cost, performance analysis, portfolio of trading strategies, hedging, risk management, risk exposure and capital allocation.

Medieval Warfare Analogy

Designing, training and equipping your army to beat a specific opponent.

A diverse strategy can handle a variety of market conditions.

A diverse strategy can handle a variety of market conditions.

3. Coding

What is it?

The method we use to build algorithmic trading strategies.

Why do we need to know it?

To build algorithmic trading strategies.

Choosing a programming language:

If you know which products you want to trade, you should find suitable brokers and platforms for these products. You then need to learn the programming languages for those platforms/backtesters.

If you are starting out and do not know what to trade, I recommend Metatrader 4 (FX and CFDs on equity indices, stocks, commodities and fixed income), Quantopian (stocks only) or Quantconnect (stocks and FX). The programming languages used are MQL4, Python and C# respectively.

Keywords to Google

Coding/programming trading strategies, MQL4, MQL5, EasyLanguage, AFL, Python/C#/C++/R/MATLAB/VBA for finance, Trading Technologies, CQG, MetaTrader 4, MetaTrader 5, Amibroker, NinjaTrader, MultiCharts, TradeStation and broker API.

Medieval Warfare Analogy

Skills to forge swords, craft bows and build catapults.

4. Data Management

What is it?

This entails sourcing and cleaning data to ensure we have accurate data for backtesting and that this data reflects the live trading environment as much as possible.

Why do we need to know it?

Garbage in == garbage out. Inaccurate data leads to inaccurate test results. We need reasonably clean data for accurate testing. Cleaning data is a trade-off between cost and accuracy. If you want more accurate data, you need to spend more resources cleaning/obtaining it.

Issues that lead to dirty data include missing data, duplicate data, wrong data (bad ticks). Other issues that can generate misleading data include dividends, stock splits, mergers, spin-offs, gaps and futures rollovers etc.

Keywords to Google

Data management, data cleaning, data providers, finance data sources, I hate cleaning equities data, data storage, data organisation and maintaining trading data.

Medieval Warfare Analogy

Making sure the intel on the enemy is correct.

5. Risk Management

What is it? (This has slight overlap with 2. Robot Design)

There are 2 main types of risk: Market risk and Operational risk. Market risk involves risk related to your trading strategy. Have you hedged away unwanted risk? Is your position sizing too large? Is your risk exposure to a particular element too high? Does it consider worst case scenarios? What if a black swan event like World War 3 happens?

In addition to managing market risk, you need to look at operational risk. System crashes, loss of internet connection, poor execution algorithm (leading to poorly executed prices or missed trades due to an inability to handle requotes), counter-party risk, broker insolvency and theft by hackers are very real issues.

Why do we need to know it?

We need to protect our downsides. Minimising risk whilst maximising returns is key.

Keywords to Google

Risk management, drawdowns, black swan events, fat-tailed events, hedging, risk exposure, operation risk, software security, secure VPS, auto restart [insert trading software’s name], downtime prevention, poor trade execution, slippage, requotes, counter-party risk and broker insolvency.

Medieval Warfare Analogy

Market risk: Reducing loopholes in the war strategy.

Operation risk: Making sure everything that is supposed to work, works.

Boom!

6. Live Execution

What is it?

Backtesting and live trading are very different. You’ll need to select proper brokers (MM vs STP vs ECN). Forexpeacearmy.com is your best friend when it comes to retail broker reviews (this also applies to brokers offering non-forex products), so make sure you read the reviews there before deciding on a broker.

You need proper infrastructure (secure VPS and downtime handling etc) and evaluation procedures (monitor your robots’ performance and analyse them in relation to market inefficiencies/backtests/optimisations) to manage your robot throughout its lifetime.

You need to know when to intervene (modify/update/shutdown/turn on your robots) and when not to.

Why do we need to know it?

The past does not predict the future perfectly. There are many issues that can crop up when trading live money. It’s essential that you create protocols to regularly monitor the market and your strategies’ performance. You also need to be prepared to improve/update your strategies and fix problems when they arise.

Keywords to Google

Live trading, trade psychology, trade management, trade execution, VPS (virtual private server), monitoring trading robots, remote access to your trading platform, live performance evaluation, trading broker selection, broker reviews, does my broker screw me over, spread widening, stop hunting, funds get deposited quickly but takes forever to withdraw, broker types, market makers, straight-through processing, electronic communication networks, A-books, B-books, liquidity pools and dark pools.

Medieval Warfare Analogy

How you manage the battle when it actually happens.

It’s a Never-Ending War!

Algorithmic trading (or any kind of trading) is a marathon, not a sprint. Keep learning and improving. The trading/investment space is getting incredibly competitive. Many strategies that used to work don’t any more. Personally, I think profitable trading systems have a lifespan of about 2-3 years (in general) before others catch on to it. You need to constantly innovate to stay ahead of the game, and innovation takes experience, wits, time, infrastructure and money.


AlgoTrading101 is an Investopedia-featured algorithmic trading course that doesn’t suck. Learn more about us at AlgoTrading101.

Keep up to date


What is Curve Fitting (Overfitting) in Trading?

Ok, yes, I understand that many of you experienced traders feel that curve fitting (aka overfitting aka data fitting) is such a rudimentary (and over-blogged) topic. However, understanding this concept is extremely important for designing and testing effective trading strategies. Thus, for those of you who are new to trading; make sure you thoroughly understand this foundational theory!

overfitting comics

Understanding Curve Fitting

Definition (from Wiki[1])

In statistics and machine learning, overfitting (curve fitting) occurs when a statistical model describes random error or noise instead of the underlying relationship. Overfitting generally occurs when a model is excessively complex, such as having too many parameters relative to the number of observations.

Definition in Simpler Words

Curve fitting is the process of adapting a trading system so closely to historical data (which includes both noise and signals) that it becomes ineffective in the future.

Why is Curve Fitting Bad?

The past does not predict the future perfectly, especially in financial markets.

Adapting strategies too closely to past data will result in an inflexibility to adapt to the future. Hence, it leads to poor performance in the future.

What Does It Mean to Reduce Curve Fitting?

We need to adapt our trading strategies to signals in historical data, not noise.

Curve Fitting in Nature

The idea of curve fitting can be seen in nature. We shall use 3 animals as case studies to understand curve fitting in nature.

giraffe2

I die easily…

Case 1: Giraffe (Curve Fitted)

Can only survive on land. Prefers leafy tall trees.

We consider giraffes as curve fitted animals. If we throw them into the ocean, they die. If we throw them into Antarctica, they die. If we remove leafy trees and just leave them with grass patches, they will find it difficult to survive.

Hence, the future of their environment needs to be similar to its past. If the giraffes’ environment starts changing aversely, they will be in trouble.

Case 2: Turtles (Reasonably Robust[2])

Can survive on land and water, but they are cold-blooded.

Turtles are much hardier creatures. However, the downside is that they are cold-blooded. Their body temperature relies on their surrounding environment. That said, these creatures can be considered reasonably robust. They can survive most changes in their environmental conditions.

Case 3: Tardigrade aka Water Bears (Extremely Robust)

Can survive in extreme temperatures [1 K (−458 °F; −272 °C) to about 420 K (300 °F; 150 °C)], pressure, radiation and outer space.

Fig 2: Look at this cutie!

Look at this cutie!

What kind of monster is this?! Tardigrades are tiny (1mm in length) but they are the hardiest animal on earth. Just a few days ago, I read that a tardigrade was brought back to life after being frozen for 30 years[3]! And it gave birth to 14 healthy babies!

Tardigrades are extremely robust. They can survive almost anything that the future decides to throw at them. In general, we should design trading strategies in the same way that nature designs tardigrades.

Data-centric View of Curve Fitting

Chart view of curve fitting

Modelling data points.

 Now that we have a better understanding of curve fitting, let’s leave the animal kingdom and head back to Wall Street. Looking at curve fitting in nature is interesting, but we need to understand the context of curve fitting in trading.

In Fig 3, we have 3 charts with the same data. We are trying to create a model that fits the shape of the data. We can clearly see that the data forms a U-shape. This model will be used to predict future data points.

In the leftmost chart, our model is a straight line. This is not representative of the data at all. It will have poor predictive abilities. We describe this model as an underfitted model.

In the rightmost chart, our model intercepts every data point. This model fits the data perfectly. On paper, this seems like the perfect model. And it is, if we are not using it to predict the future. You see, unless future data points follow the past perfectly, this model will have very poor predictive value. This is an overfitted model.

In the middle chart, our model describes the general shape of the data points. This model does have some level of error – it does not intercept all the data points. However, this is fine. We need our models to have a certain degree of error. This means that the model does not rigidly follow the past. This model captures the signal in the data (signal refers to the U-shaped data points). Thus, it should be able to adapt to minor changes to the data structure in future. This model is a good fit aka it is robust.

Demonstrating Curve Fitting

Enough theory! Let’s see some action! Ok sir, let’s curve fit some stuff on purpose.

In this exercise, we will curve fit a basic trading robot we use at AlgoTrading101 called Belinda.

Disclaimer: This is the WRONG way to conduct your optimisation[4]. Do not try this at home!

Step 1:

We run an optimisation for Belinda by varying 3 variables: sma_short, sma_long and atr_period.

1

Step 2:

We run the optimisation from 1st April 2014 to 1st January 2015.

2

Step 3:

We find the optimised parameter values aka the parameter values that produce the best objective function.

3

Step 4:

Using the optimised parameter values, we run a backtest[5] to see the performance and equity curve in detail. We use the same backtest dates as before: 1st April 2014 to 1st Jan 2015. We should expect to see a profit of $3,549.18.

4

Step 5:

Now we test Belinda with the optimised parameter values using data from the future. As mentioned, the future rarely reflects the past perfectly. Thus, we do not expect this backtest to be profitable.

Let’s run the backtest in the future period: 1st Jan 2015 to 1st Oct 2015 (the next 9 months after previous period).

5

What a disaster! (Unsurprisingly)

Ok, now that we know the negative effects of curve fitting, the next question is “How do we minimise curve fitting?” At AlgoTrading101 we have designed 10 methods to reduce curve fitting. Unfortunately, that is paid content so if you are keen, go sign up!

Before you go…

We need to show you something called performance manipulation. Occasionally, you may see some people selling trading robots over the web. They claim that their robot can make 100% returns overnight. And they manage to produce some equity charts and performance tables that do indeed indicate 100% returns in a night.

So, is their robot legitimate? Well, I have never bought such robots so I can’t refute their claim. However, I may have some insights into how they produce such incredible performance.

Step 1:

Run an optimisation and find the optimised parameter values. Run a backtest using these parameter values and the same dates as used in the optimisation. The difference now is that we increase our position size to insane levels. We use 20% risk per trade.

Short version: Repeat Step 4 in the above example but trade much bigger!

6

Look at that performance! Did we just turn $10,000 into $269,086 over 9 months?!

7

Oops, it is not $269,086. It is $2,544,741.19! I’m going to be a zillionaire!!! (And yes, zillionaire is a real word!)


AlgoTrading101 is an Investopedia-featured algorithmic trading course that doesn’t suck. Learn more about us at AlgoTrading101.

Keep up to date


[1] https://en.wikipedia.org/wiki/Overfitting

[2] When we use the word robust in this post, we are referring to Optimisation Robustness (i.e. Maximising objective function while minimising curve fitting.)

[3] http://www.sciencealert.com/a-frozen-tardigrade-has-been-brought-back-to-life-after-30-years

[4] http://www.investopedia.com/terms/o/optimization.asp

[5] http://www.investopedia.com/terms/b/backtesting.asp

 

Dynamic Trading Systems: Fixed Strategies Can’t Adapt!

“We need our trading strategies to adapt to the market to maintain their effectiveness!”

What does that statement mean and how do we achieve it? For our trading strategies to adapt to the markets, they need to have adaptive components.

Defining Adaptive Components

Adaptive Component: A component in the trading strategy that adapts to market conditions.

Adaptive components could be in any part of the trading strategy – entries, exits and position sizing e.g. signal generating rules, indicators, take profit and stop loss etc). Let’s look at some examples to get a clearer picture.

Example of Non-Adaptive Components (Not Good!)

  • Hard stop of 100 pips
  • Enter a trade when market is 50 pips above yesterday’s high
  • Enter 1.5 standard lots per trade

Example of Adaptive Components (Good!)

  • Hard stop of 2 ATR (adapting to volatility)
  • Enter a trade when market breaks 20-day high (adapting to market range[1])
  • Risk 1% per trade (adapting to account balance)

Importance of Adaptive components

We want our trading strategies to adapt to market conditions to maintain their efficiency. Efficiency is defined as the ability to capture market inefficiencies.

Let’s look at a case study of a component adapting to market range:

non-adapt chart

Fig 1: Non-adaptive mean-reversion strategy.

In the scenario above, we have a non-adaptive entry rule. We short at the top red line and long at that bottom red line. This rule works well between time = 0 and time = 1. As the market behaviour changes, the strategy is unable to adapt.

The aim of this strategy is to capture mean-reversion tendencies, but it fails to do that between time = 1 to time = 3. Between time = 1 and time = 2 it does not capture any trades. Between time = 2 and time = 3 it has a tendency to enter some of the trades too early.

adaptive chart

Fig 2: Adaptive mean-reversion strategy.

In this new (ideal) scenario, the entry rules are changed such that they adapt to the market range. In every time period it changes itself to adapt to the new market range.

What Should We Adapt To?

Main Priorities

Our main priority is to adapt to the market inefficiency we are capturing.

Examples:

  • On a mean-reverting trade, our entry/exit rules must adapt to market range.
  • If your strategy depends on tweets, your rules may need to adapt to the rate of tweets or overall tweet quantity/ratio.
  • If you trade co-integration between stocks, you may want to find out how the extent of price discrepancies affects your strategy’s performance (linearly or exponentially etc) and design your rules accordingly.

Secondary Priorities

Besides the market inefficiency, there are other elements to adapt to. What these elements are depends on our trading strategies’ characteristics. There isn’t a one-size-fit-all list for this but here are some popular ones.

Examples:

  • Volatility – Few strategies are volatility independent. A common component that should adapt to volatility is our position sizing.
  • Account Balance – Trade less when your account is smaller and vice versa! (Unless it’s a martingale betting strategy, which is generally a bad idea)
  • Current Risk Exposure – Total amount you can lose if all your positions get stopped out. We need to find the sweet spot.

Adapting Our Adaptive Component

We need to go one step deeper in order to adapt. Adaptive components may have fixed components within them, so we need to adapt our adaptive components. To illustrate this, let’s look at some strategy rules.

Case Study:

  • Enter on 20-day high
  • Stop Loss of 2*ATR(15)

What determines these fixed parameter values: 20, 2 and 15?

In order to adapt our parameter values, we need to conduct a special type of optimisation called the Walk-Forward Optimisation (WFO). However, we won’t be talking about it in detail here.

For now, just know that the WFO enables us to forecast optimal future parameter values using past data, with minimal curve fitting if done correctly.


AlgoTrading101 is an Investopedia-featured algorithmic trading course that doesn’t suck. Learn more about us at AlgoTrading101.

Keep up to date


[1] Price levels that the market is trading within.

 

1

Trading Edge: 4 Types of Market Inefficiencies

Generating trading ideas can be a frustrating process, especially if there isn’t a structured framework for it. To assist us with our ideas generation process, we break down algorithmic trading ideas into 4 main categories. These 4 categories are essentially different types of market inefficiencies. As mentioned in the previous blog post, market inefficiencies are the core building blocks of trading strategies.

The 4 types are:

  • Fundamentals
  • Statistics
  • Macroeconomics
  • Market Microstructure

Note that these 4 types are not mutually exclusive. Strategies can involve more than 1 type.

simpsons_begin

Understanding Market Inefficiencies

Fundamentals

The term fundamentals is commonly associated with equities. In this case, we use fundamentals to describe the value stemming from the intrinsic make up of any asset. These include earnings releases, oil field discoveries and expected downgrades of corporate bonds.

Examples of fundamental inefficiencies/opportunities:

  • Overstated earnings in a company’s annual report.
  • A major breakthrough in clean energy technology disrupting traditional oil industries.
  • Death of a superstar CEO.

Example of algorithmic trading strategy:

We create an algorithm that predicts company earnings based on publicly available information. Let’s say we’re looking at a timber logging company.

We input information regarding revenue and cost such as amount of land deforested, number of new contracts secured, market price of timber, amount of new equipment purchased, labour and overhead costs etc. The algorithm outputs the expected earnings of the company for the quarter and we compare that with analysts’ earning forecasts. If there is a significant discrepancy, we put in a position before the earnings release and exit the position after.

We do this for thousands of companies to lower the variance in our performance.

Statistics

If we break it down to first principles[1], strategies based on statistics essentially say “This market move is unusually big or small for this market condition.”. How unusual this is considered is defined by how the asset has performed historically. Of course, past performance may not be an accurate predictor for the future. Hence, we need to think critically and prudently when we implement strategies involving statistical inefficiencies. Most strategies involving statistics employ either cointegration[2], mean-reversion[3]or correlation/prediction

Examples of statistical inefficiencies/opportunities:

  • A group of stocks in the same country and industry expected to behave in a similar fashion during certain periods.
  • Corporate bond rating correlated with the stock price of some of its major clients.
  • Prediction of a timber company share price change during earnings release using publicly available information (see previous example under Fundamentals).
  • Stock A’s price correlated with the number of tweets about its impending earnings disaster.

Example of algorithmic trading strategy:

A group of stocks in the utilities industry in the same country with a similar corporate make up and market capitalisation are expected to behave in the same way during a quiet period (defined as a period where there are no expected material news/announcements).

Hence, when stock X’s price diverges from the rest in a statistical fashion, we expect the relative difference in price between the stock X and the rest to converge 8 times out of 10. We put in a trade to exploit this.

Macroeconomics

Macroeconomic inefficiencies are market inefficiencies that stem from macroeconomic events (you don’t say!). Such events include: central bank announcements/activities, economic policy shifts, government corruption and multi-country trade agreements.

Examples of macroeconomic inefficiencies/opportunities[4]:

  • Lagging reaction from a particular derivative after Non-Farm Payrolls.
  • Impact of economic sanctions on certain industries and asset groups.
  • Expectations of easing or tightening policies.

Example of algorithmic trading strategy:

A high frequency firm uses powerful computers and a lightning fast connection to get information on country A’s central bank rate cut decision. They then buy the country’s bond futures before anyone else.

Market Microstructure

According to Wikipedia[5]:

Market microstructure is a branch of finance concerned with the details of how exchange occurs in markets. The major thrust of market microstructure research examines the ways in which the working processes of a market affects determinants of transaction costs, prices, quotes, volume, and trading behaviour.

In simpler terms: It is the underlying mechanisms that enable trading in the financial markets.

You may be wondering why market microstructure is considered a source of value. Even though market microstructure seems to deal with trading infrastructure rather than the assets itself, it creates many opportunities that traders can exploit. Market microstructure inefficiencies usually involve analysing other market players or exploiting infrastructural flaws.

Examples of market microstructural inefficiencies/opportunities:

  • A fund wants to buy a large amount of stock A. They enter their trades in a manner that unintentionally signals their objective. Other market players catch on and start buying the stock too.
  • Spoofing[6] the depth of market order book to mislead other traders.
  • Out queuing other players during futures rollover in a First In, First Out exchange.

Example of algorithmic trading strategy:

While analysing the limit order book[7], we see a pattern of queuing that seems to indicate that a buyer wants to long large amount of US 10 Year futures. We design an algorithm to identify such patterns and trade accordingly.

Starting the Idea Generation Process

In addition to having a structured thought process for discovering trading ideas, we need 2 other components in the idea generation process.

First, we need to know what types of strategies are suitable for the trader. Factors to consider include: trading capital, risk profile and programming competency.

Second, we need a framework to vet trading ideas and select promising ones for our backtesting stage. However, we won’t be going into details for those 2 components today. To learn more, check out our course, AlgoTrading101. That’s it for today’s post!


AlgoTrading101 is an Investopedia-featured algorithmic trading course that doesn’t suck. Learn more about us at AlgoTrading101.

Keep up to date


 

[1] https://en.wikipedia.org/wiki/First_principle

[2] https://en.wikipedia.org/wiki/Cointegration

[3] http://www.investopedia.com/terms/m/meanreversion.asp

[4] Not all of these inefficiencies are suitable for algorithmic trading.

[5] https://en.wikipedia.org/wiki/Market_microstructure

[6] https://en.wikipedia.org/wiki/Spoofing_(finance)

[7] https://en.wikipedia.org/wiki/Central_limit_order_book

1

Market Prudence – Reason For The Trade

Approach to Designing Amazing Strategies

“Add an SMA(30)! No, add an EMA(18)! Actually, maybe it’s time to optimise it to find the best parameter value. Ok, we’ll stick to EMA(15). Throw in 3 date and time filters, 3 optimised price indicators and 3 volume indicators (that are essentially saying the same thing in different ways) and we’ll surely have the ultimate trading strategy!”

The above scenario describes a disaster waiting to happen. When we design trading strategies at AlgoTrading101, we break everything down to the basic building blocks. This allows us to build logical and effective trading systems. This means that we do not over-optimise parameters and add layers of indicators/rules until the cows come home.

It is very important to design strategies in a market prudent manner. This is a foundational step towards building long term profitable trading systems.

What is Market Prudence

A trading system is said to be Market Prudent if its underlying logic exploits a fundamental market inefficiency.

This fundamental market inefficiency is the reason for our trades. This is the basic building block for trading strategies. It applies both to manual and automated trading.

Examples of such inefficiencies are:

  • Mean reversion of a currency pair due to similar macroeconomic factors in two countries.
  • Price discrepancies (after considering currency disparity and transaction cost) between the same assets that are trading on different exchanges.
  • Mispricing of structured assets due to misinformation.

Market Prudent Approach to Design Strategies

There are 2 steps for a market prudent approach:

Step 1: Define the fundamental inefficiency that we are trying to capture

Step 2: Create a set of rules to capture this inefficiency

Let’s look at a fictional example:

Step 1: US Dollar Index is very trendy the week after Non-Farm Payrolls (NFP) because of macroeconomic reasons. However, people tend to over-speculate and this causes prices to retrace if they jump too much.

Step 2: Create a strategy that captures trends and implement it post NFP. However, we still need to add something to cut positions if the move is exaggerated (exaggerated is defined both statistically and from a market prudent point of view). Hence, we add an oscillator or a statistical indicator in our exit rules that identifies exaggerated moves.

Now that we know our reason for the trade, if our inefficiency ceases to exist (sudden major ECB announcement, World War III happens etc), we can readjust/pause our strategy – this is done immediately and not after losses occur. Every decision is based on a fundamental premise, not unfounded guesswork or gambles.

Non-Market Prudent Approach to Design Strategies

Steps for non-market prudent approach: I’m not sure what they are, it varies from trader to trader.

Let’s look at a fictional example:

Step 1: Trader looks at chart, finds patterns. CADUSD seems to trend a lot on the 2 week of the month.

Step 2: Throw in some EMAs, RSI, CCI and Volume Indicators.

Step 3: Add a day filter, test a no-trade rule on all 5 weekdays. If we don’t trade on Mondays, we get higher returns. Let’s not trade on Mondays (there may be fundamental reasons not to trade on Monday but this approach fails to identify it).

Step 4: Run brute force optimisation with no walk-forward. Ok we found the best values. EMA(20) works best. EMA(25) completely destroys the strategy. Hence the key ingredient to our strategy is in using EMA(20) and not EMA(25).

Strategy either succeeds in the short run or flops. Either way, we don’t know why and can’t take logical steps to manage our strategies and protect ourselves. Everything is pseudo logic at best, an absolute gamble at worst.

Simplicity is Key

Everything should be made as simple as possible, but no simpler.  – Albert Einstein

We only add what is needed, but not more. Every additional layer of rules increases the strategies complexity exponentially. Extra rules mean that our strategies become more rigid and this makes them more susceptible to failure.

When our strategies are simple and market prudent, we can understand them well. Understanding our strategies is very important for designing long term profitable systems. We need to know when and why our strategies work or break down.

TL; DR/Summary – Design simple trading strategies that exploits fundamental market inefficiencies.

 

In the next blog post, we’ll talk about the 4 main types of trading strategies. Drop us your email below to stay in the loop!


AlgoTrading101 is an Investopedia-featured algorithmic trading course that doesn’t suck. Learn more about us at AlgoTrading101.

Keep up to date

2

Impossible Trinity Of (Position) Sizing

What is the Impossible Trinity

When we think about the number of lots to trade (position sizing), we may face an interesting situation called the Impossible Trinity of Sizing. The Impossible Trinity of Sizing is a trilemma which states that it is impossible for us to control over all 3 of the following at the same time:

  • Stop Loss Size
  • Risk per Trade
  • Leverage Used
Impossible Trinity Of Sizing

Impossible Trinity Of Sizing

The Impossible Trinity of Sizing states that we can only control at most two of the three factors – Stop Loss Size, Risk per Trade and Leverage Used.

This trilemma exists when we use an account risk position sizing method. To recall, the account risk method calculates our trade size according to the percentage of our account balance that we are willing to risk. Hence, if we have a $100,000 account and we plan to risk 2% per trade, we need to trade a certain quantity such that we lose $2,000 in one trade.

Formula/Sizing Algorithm:

Quantity per Trade (in $) = (Risk per Trade * Account Balance) / Stop Loss Size

Quantity per Trade (in Lot) = (Risk per Trade * Account Balance) / (Stop Loss Size * Contract Size)

Application to Trading

Let us examine what happens when we try to control all three factors. Here’s our setup:

  1. Trading EURUSD
  2. Account balance of 100,000USD
  3. Risk 2% per Trade (Determined by us)
  4. Stop Loss Size of 100 pips (Determined by us)
  5. Leverage Used (Undetermined)

Based on the above formula, we trade 2 lots. Leverage Used is 2 to 1 (Trading 2 lots requires $200,000). We decided to control Risk per Trade and Stop Loss Size in this example and Leverage Used was decided for us.

We can see that it is not possible to control Leverage Used in this situation. If we want to control our leverage, we need to modify the values of Risk per Trade or Stop Loss Size to ensure our Sizing Algorithm functions accurately.



Selecting Two Corners

The million dollar question is: Which two factors should we control?

Scenario 1: Controlling Stop Loss Size and Sizing

Stop Levels and Sizing should be strategy-dependent. Thus, our first choice is to control Stop Loss Size and Sizing.

Problems arises when our broker does not give us enough leverage for our trades. In this case, we need to decide if we want to give up control on Stop Loss Size or Sizing. In order to decide this, we must understand which factor is more important for our robot’s performance.

Scenario 2: Controlling Stop Loss Size and Leverage Used

Giving up control over sizing is synonymous to leaving your money management strategy to your broker. Fortunately, this may decrease your upside but will not increase your downside per trade.

Let us assume that our broker gives us a maximum leverage of 1.5 to 1. In our previous example, we need a leverage of 2 to 1. If we fix our Leverage Used to the maximum amount (1.5 to 1), our Risk per Trade drops to 1.5%.

If our broker gives us a maximum leverage of more than 2 to 1, our Risk per Trade will stay at 2%. Our broker cannot force us to increase our Risk per Trade by increasing maximum leverage. Hence, the amount we risk (downside) will not increase.

Scenario 3: Controlling Sizing and Leverage Used

At AlgoTrading101, we believe that Stop Loss Size is a tool to buffer market noise. Hence, giving up control of Stop Loss Size is usually a bad idea. However, there are times when retaining control of Sizing is more important than Stop Loss Size.

For example, strategies such as martingale betting (not recommended) require a specific sizing structure to work. When faced with such strategies and leverage limitations, we opt to give up control of Stop Loss Size.

Bottom Line

This article serves to explore the effects of leverage limitations. Traders need to be aware of the impact these constraints have on a robot’s architecture in order to design long term profitable strategies.


AlgoTrading101 is an Investopedia-featured algorithmic trading course that doesn’t suck. Learn more about us at AlgoTrading101.

Keep up to date