Skip to main content

Main menu

  • Home
  • Current Issue
  • Past Issues
  • Videos
  • Submit an article
  • More
    • About JOI
    • Editorial Board
    • Published Ahead of Print (PAP)
  • IPR Logo
  • About Us
  • Journals
  • Publish
  • Advertise
  • Videos
  • Webinars
  • More
    • Awards
    • Article Licensing
    • Academic Use
  • Follow IIJ on LinkedIn
  • Follow IIJ on Twitter

User menu

  • Sample our Content
  • Request a Demo
  • Log in

Search

  • ADVANCED SEARCH: Discover more content by journal, author or time frame
The Journal of Investing
  • IPR Logo
  • About Us
  • Journals
  • Publish
  • Advertise
  • Videos
  • Webinars
  • More
    • Awards
    • Article Licensing
    • Academic Use
  • Sample our Content
  • Request a Demo
  • Log in
The Journal of Investing

The Journal of Investing

ADVANCED SEARCH: Discover more content by journal, author or time frame

  • Home
  • Current Issue
  • Past Issues
  • Videos
  • Submit an article
  • More
    • About JOI
    • Editorial Board
    • Published Ahead of Print (PAP)
  • Follow IIJ on LinkedIn
  • Follow IIJ on Twitter

All That Glitters Is Not Gold: Comparing Backtest and Out-of-Sample Performance on a Large Cohort of Trading Algorithms

Thomas Wiecki, Andrew Campbell, Justin Lent and Jessica Stauth
The Journal of Investing Fall 2016, 25 (3) 69-80; DOI: https://doi.org/10.3905/joi.2016.25.3.069
Thomas Wiecki
is the data science lead at Quantopian in Boston, MA.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • For correspondence: twiecki@quantopian.com
Andrew Campbell
is a data scientist at Quantopian in Boston, MA.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Justin Lent
is the director of hedge fund development at Quantopian in Boston, MA.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Jessica Stauth
is the vice president of quant strategy at Quantopian in Boston, MA.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • For correspondence: jstauth@quantopian.com
  • Article
  • Info & Metrics
  • PDF (Subscribers Only)
Loading

Click to login and read the full article.

Don’t have access? Click here to request a demo 

Alternatively, Call a member of the team to discuss membership options
US and Overseas: +1 646-931-9045
UK: 0207 139 1600

Abstract

When automated trading strategies are developed and evaluated using backtests on historical pricing data, there exists a tendency to overfit to the past. Using a unique dataset of 888 algorithmic trading strategies developed and backtested on the Quantopian platform, with at least six months of out-of-sample performance, this article studies the prevalence and impact of backtest overfitting. Specifically, the authors find that commonly reported backtest evaluation metrics, such as the Sharpe ratio, offer little value in predicting out-of-sample performance (R 2 < 0.025). In contrast, higher-order moments, such as volatility and maximum drawdown, as well as portfolio construction features (e.g., hedging), show significant predictive value of relevance to quantitative finance practitioners. Moreover, in line with prior theoretical considerations, the authors find empirical evidence of overfitting—the more backtesting a quant has done for a strategy, the larger the discrepancy between backtest and out-of-sample performance. Finally, they show that by training nonlinear, machine-learning classifiers on a variety of features that describe backtest behavior, out-of-sample performance can be predicted with much greater accuracy (R 2 = 0.17) on hold-out data than when using linear, univariate features. A portfolio constructed by using predictions on hold-out data performed significantly better out-of-sample than one constructed from algorithms with the highest backtest Sharpe ratios.

TOPICS: Statistical methods, portfolio construction, portfolio theory

  • © 2016 Pageant Media Ltd
View Full Text

Don’t have access? Click here to request a demo

Alternatively, Call a member of the team to discuss membership options

US and Overseas: +1 646-931-9045

UK: 0207 139 1600

Log in using your username and password

Forgot your user name or password?
PreviousNext
Back to top

Explore our content to discover more relevant research

  • By topic
  • Across journals
  • From the experts
  • Monthly highlights
  • Special collections

In this issue

The Journal of Investing: 25 (3)
The Journal of Investing
Vol. 25, Issue 3
Fall 2016
  • Table of Contents
  • Index by author
Print
Download PDF
Article Alerts
Sign In to Email Alerts with your Email Address
Email Article

Thank you for your interest in spreading the word on The Journal of Investing.

NOTE: We only request your email address so that the person you are recommending the page to knows that you wanted them to see it, and that it is not junk mail. We do not capture any email address.

Enter multiple addresses on separate lines or separate them with commas.
All That Glitters Is Not Gold: Comparing Backtest and Out-of-Sample Performance on a Large Cohort of Trading Algorithms
(Your Name) has sent you a message from The Journal of Investing
(Your Name) thought you would like to see the The Journal of Investing web site.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Citation Tools
All That Glitters Is Not Gold: Comparing Backtest and Out-of-Sample Performance on a Large Cohort of Trading Algorithms
Thomas Wiecki, Andrew Campbell, Justin Lent, Jessica Stauth
The Journal of Investing Aug 2016, 25 (3) 69-80; DOI: 10.3905/joi.2016.25.3.069

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
Save To My Folders
Share
All That Glitters Is Not Gold: Comparing Backtest and Out-of-Sample Performance on a Large Cohort of Trading Algorithms
Thomas Wiecki, Andrew Campbell, Justin Lent, Jessica Stauth
The Journal of Investing Aug 2016, 25 (3) 69-80; DOI: 10.3905/joi.2016.25.3.069
del.icio.us logo Digg logo Reddit logo Twitter logo Facebook logo Google logo LinkedIn logo Mendeley logo
Tweet Widget Facebook Like LinkedIn logo

Jump to section

  • Article
    • Abstract
    • METHODS
    • RESULTS
    • DISCUSSION
    • LIMITATIONS
    • APPENDIX
    • ENDNOTES
    • REFERENCES
  • Info & Metrics
  • PDF (Subscribers Only)
  • PDF (Subscribers Only)

Similar Articles

Cited By...

  • Crowdsourced Investment Research Through Tournaments
  • Google Scholar
LONDON
One London Wall, London, EC2Y 5EA
United Kingdom
+44 207 139 1600
 
NEW YORK
41 Madison Avenue, New York, NY 10010
USA
+1 646 931 9045
pm-research@pageantmedia.com
 

Stay Connected

  • Follow IIJ on LinkedIn
  • Follow IIJ on Twitter

MORE FROM PMR

  • Home
  • Awards
  • Investment Guides
  • Videos
  • About PMR

INFORMATION FOR

  • Academics
  • Agents
  • Authors
  • Content Usage Terms

GET INVOLVED

  • Advertise
  • Publish
  • Article Licensing
  • Contact Us
  • Subscribe Now
  • Log In
  • Update your profile
  • Give us your feedback

© 2022 Pageant Media Ltd | All Rights Reserved | ISSN: 1068-0896 | E-ISSN: 2168-8613

  • Site Map
  • Terms & Conditions
  • Cookies
  • Privacy Policy