Back-Tested Strategies: Real or Random?

Have you ever seen a bad back-test?

Investment professionals have been jokingly asking that question for years, and the answer remains the same: of course not. That is because no one will likely visit your office to discuss a new product designed to be smart beta, strategic beta, or what I’ll call factor-based whose simulated history only offers mediocre performance.

Why? Because few would buy it.

Which brings us to how investment products are (often) made and how you can determine whether they are worth your clients’ money.

Factor-based products are often developed when asset managers examine historical data to try to determine what attributes of securities may have driven outperformance over time. Before a product is launched, a rules-based methodology may be implemented and applied to historical data as if the methodology had begun earlier, hence the term back-tested. What one person might consider research another might call data mining, and there can be a fine line between the two. But however you think about it, there is a difference between finding a random anomaly and identifying a viable rules-based strategy.

As a fun example of this, a few years ago my colleagues Joel Dickson and Chuck Thomas ran a hypothetical simulation that compared the performance of the S&P 500 Index with an equity portfolio that had an equally weighted combination of all stocks with tickers that began with S, M, A, R, or T. As the figure below shows, this simple, rules-based strategy did very well over a long period of time. However, let’s be honest, there is no sound reason to justify why it would be a good idea to pursue this strategy in the future.

Annualized return of S.M.A.R.T. beta strategy from December 31, 1994, to October 31, 2013

Source: Vanguard.

Note: The S.M.A.R.T. beta strategy is hypothetical in nature and does not represent the returns of any Index or Investment vehicle. It is constructed with equal-weighted components of all current securities in the S&P 500 whose tickers begin with the letters S.M.A.R.T. and rebalanced monthly.

While in most cases, it is hard to eliminate the risks of data mining entirely, there are a number of ways to help improve your confidence in the potential of a simple rules-based strategy to produce a return premium for a client in the future.

I invite you to download our one-page checklist for evaluating back-tested strategies, which identifies common biases that can occur when products are created. The questions in the checklist are ones you may want to ask any time you are considering a factor-based product.

These details are grouped under three overarching questions:

1. Is there an enduring, sensible rationale?

Consider whether there is a risk-based reason for the strategy. If there is, then there may be some intuition as to why the strategy may persist with rational asset pricing (in this scenario, no market inefficiency is needed). If the reason is that some investors believe other investors have made and will continue to make a similar mistake that creates an anomaly, then a sound reason should be given as to why some investors will continue to make that error and/or why other investors do not take full advantage of the mistake.

2. Does extensive evidence support the strategy?

In other words, what quantitative evidence supports this strategy generating a return premium? Geeks like me often call this robustness testing. This means taking a rigorous approach to looking at the historical data through advanced econometric techniques. There is a lot of jargon in this area of finance (terms such as out-of-sample tests, overfitting bias, and multiple testing bias); however, I’ll spare you from it here. In the checklist, we listed some questions that simplify the technical aspects so you can ask managers/analysts the right questions.

3. Will the strategy work after all costs are considered?

This is often an underappreciated aspect of due diligence. Investors must believe these return premiums will survive real-world implementation costs (e.g., trading commissions, bid-ask spreads, management expenses, and taxes), which can really add up over time. The actual significance of these various effects can be influenced by numerous issues, such as the specific rules-based strategy, the way securities are weighted, the size of the potential investment, the rebalancing policy, the type of vehicle chosen for implementation (e.g., ETF or conventional mutual fund), and the circumstances of the client under consideration.

In the coming weeks or months, you will no doubt either see an advertisement about or hear a sales person pitch a rules-based strategy. Just remember the importance of thoroughly evaluating whether the efficacy of a strategy that may have worked in the past is real or just random.

I would like to thank my colleague Tom Paradise for his contributions to this blog post.

Download our one-page checklist for evaluating back-tested strategies.

Doug Grim

Doug Grim, CFA, is a senior investment strategist in Vanguard Investment Strategy Group, where he leads the team that conducts research and provides thought leadership on issues related to factor-based portfolio construction and due diligence for investors. Before his current role, he was a senior investment consultant in Vanguard Institutional Advisory Services^®. In that position, he provided asset allocation and portfolio construction recommendations, investment policy consulting, and capital markets research to institutional clients. He also served as team leader responsible for assisting other consultants with all asset allocation and asset/liability modeling studies conducted for clients.

Mr. Grim earned a B.S. from the University of North Carolina at Wilmington. He is a CFA^® charterholder and a member of the CFA Institute and the CFA Society of Philadelphia.

View more posts by Doug Grim

For more information about Vanguard funds or Vanguard ETFs, visit advisors.vanguard.com or call 800-997-2798 to obtain a prospectus or, if available, a summary prospectus. Investment objectives, risks, charges, expenses, and other important information are contained in the prospectus; read and consider it carefully before investing.

Vanguard ETF Shares are not redeemable with the issuing Fund other than in very large aggregations worth millions of dollars. Instead, investors must buy and sell Vanguard ETF Shares in the secondary market and hold those shares in a brokerage account. In doing so, the investor may incur brokerage commissions and may pay more than net asset value when buying and receive less than net asset value when selling.

Investments in bonds are subject to interest rate, credit, and inflation risk.

Diversification does not ensure a profit or protect against a loss.

Investments in stocks or bonds issued by non-U.S. companies are subject to risks including country/regional risk and currency risk. These risks are especially high in emerging markets.

All investing is subject to risk, including possible loss of principal.

Vanguard Marketing Corporation, Distributor of the Vanguard Funds.