In this issue:
I. Commentary by Marge Sherald, COO
II. Message from Denham Ward
III. Denham Ward offers solutions to over fitting models
IV. Dennis Voigt transforms ChaosHunter model with NeuroShell Trader into a powerful system that works on different instruments and time frames
V. Save 20% on Trader Upgrades to Power User and ChaosHunter
VI. Office closed for Memorial Day, May 28, 2012
VII. One way to stop this newsletter
***********************************************************
I. Commentary by Marge Sherald, COO
This newsletter is a departure from our usual short topics, but I think you’ll agree the following articles are well worth your consideration. Denham Ward talks about preventing over fitting so models work on future data, while Dennis Voigt presents a very convincing case for using ChaosHunter and NeuroShell Trader together.
***********************************************************
II. Message from Denham Ward
I would like to thank everyone for their condolences and heartfelt sympathy regarding the passing of my father, Steve Ward.
For those that don’t know me, I thought I should provide a brief background of my involvement with Ward Systems Group. I started with Ward Systems Group over 20 years ago when I programmed the NeuroShell 2 Digital Signal Processing hardware acceleration board over one of my college summer breaks. After graduating with a Master’s Degree in Electrical Engineering, I continued working for Ward Systems Group part time while working at the University of Colorado applying neural networks in medical and anesthesiology research. In the early 90’s, my father and I sketched out plans for what was to become known as NeuroShell Trader on the back of a napkin over a long lunch. Since then I have been working full time with Ward Systems Group as the lead developer of NeuroShell Trader.
Although my father will be missed, I’d like to reassure our loyal customers that we will be moving forward in his footsteps, supporting existing products, and continuing Ward Systems Group’s history of innovations.
***********************************************************
III. Denham Ward offers solutions to over fitting models
The holy grail of trading system development is being able to create a trading system that performs into the future as well as it did in the past. Unfortunately the pitfall that most new traders fall into is tweaking a system’s rules and parameters until it shows the absolute maximum profit possible over past data. When they go to trade the system, it no longer performs as it did in the past and they lose money. In technical terms, this is called “over fitting the data”. Any time you create a trading system, see how it performed on past data, and then tweak the system to make it perform better on that same past data, you are over fitting the data. The problem is compounded when using software like NeuroShell Trader which optimizes by evaluating hundreds, thousands or even millions of different parameter and rule combinations. However, by following a few simple rules, using advanced features found in NeuroShell Trader and utilizing a new feature of the Power User version, you can reduce over fitting and develop more robust trading systems.
The first thing to recognize is that the more rules and numeric parameters a system has, the greater the chance of over fitting the data. Take for example the simple rule “go long at 10 am and exit at 3 pm”. Such a simple system doesn’t over fit the data and is just as likely to lose money as it to make money on any given day. Now for illustration purposes, let’s say you look at the past day of trading and notice the system could have made a lot more money by making the rules long at 9:38 am, short at 10:33 am, long at 10:54 am, short at 11:36 am, long at 1:12 pm, short at 3:48 pm and then exit at 3:57 pm. You can see how adding more rules and parameters over fits the system so it will only perform well on days that have the same price movements as the data set you used to tweak the system. Whether a system is based on time entries, a Bollinger band system or a complex neural network analysis, using fewer rules and numeric parameters in a system minimizes the chance of over fitting.
As a rule of thumb, the trading system that has the absolute maximum profit possible over a past data set is most likely over fitting the past price movements. It is better to choose a system that doesn’t maximize the overall profit, but instead has a smoother equity curve with smaller drawdowns which gradually grows towards a modest profit. In NeuroShell Trader, the decision of what is the “best” system during optimization is controlled by the Objective Function. In NeuroShell Trader 6.0 even more objective functions were added to allow greater control over the optimization results. So even though the traditional “Maximize Return on Account” or “Maximize Profit” may look the best on past data, try using an old favorite like “Maximize Return On Account * Equity Curve Correlation” or one of the new 6.0 objective functions like “Maximize Ulcer Performance Index” to avoid over fitting.
How you use past data also contributes greatly to the robustness of a trading system. If a system is tweaked and evaluated on a single data set, it won’t take long until it is over fit on that data. A better method is to tweak the system on one data set. Once you’re satisfied with the system, apply it to a second independent data set to evaluate how well it would have performed had you traded the 2nd data set. The problem is that if each time a new tweak is found that does better on the 1st data, the 2nd data set is used to choose between the new best and old best tweak, then the 2nd data set is now being over fit and a 3rd independent data set is needed for truly evaluating future trading. Fortunately, unlike many trading software products on the market, NeuroShell Trader can optimize systems with up to 3 different data sets as described above. On the Dates tab of the trading strategy and prediction wizard, you can setup the gray “optimization”, orange “paper trading” and green “trading” data sets which correspond to the 1st, 2nd and 3rd data sets described above. Using the gray optimization and orange paper trading should reduce over fitting during optimization while the green trading data still provides an evaluation of how the final system will do in future trading.
Although the methods described above are good at reducing data over fitting, it is the walk-forward optimization feature of the new Power User versions of NeuroShell Trader that provides the ultimate tool for evaluating how well an optimized trading system will do in the future. In a nutshell, walk-forward optimization can answer a question like “How much money would I have made trading this system over the past two years, but re-optimizing the system rules and parameters every month to account for changing market conditions?” To answer this question in the Power User version, set the green trading set size to 1 Month, choose walk forward optimization with 24 walk forwards (12 months per year * 2 years) and select “Carry forward actual trading account balances to next walk-forward”. NeuroShell Trader first optimizes on data prior to 24 months ago based upon your desired optimization and paper trading sizes and then applies the optimized rules and parameters to month 24 for green trading statistics. The data sets are then be moved forward by 1 month, the system re-optimized and applied to month 23 green trading statistics. The process continues until the data sets have been moved forward 24 times, 24 optimizations have occurred and the corresponding 24 months of green trading statistics are created. When NeuroShell finishes, the sum of the 24 green trading statistic profits indicates how much money would have been made or lost trading for 2 years while re-optimizing every month. Since the green trading statistics are totally independent of the data used for deciding upon the 24 different set of rules and parameters, the cumulative sum of the walk-forward optimizations provides the ultimate evaluation for data over fitting and future performance.
Although unpredictable world events and news will always be a threat to even the best trading system, you’ll have a fighting chance when you minimize your systems rules and parameters, choose your objective function wisely, use multiple data sets when optimizing, and then use walk-forward optimization as the ultimate test of future trading performance.
***********************************************************
IV. Dennis Voigt transforms ChaosHunter model with NeuroShell Trader into a powerful system that works on different instruments and time frames
Dennis Voigt is the developer of the BMJ Fractal Analysis Indicator Set for NeuroShell Trader. His indicators can produce a filtered time series that is theoretically stripped of most of the noise in the data. The result is a highly responsive smoothing filter of any data series. He writes about how he combines his indicators in a successful ChaosHunter trading formula that he further enhances in NeuroShell Trader. A more detailed version of this article is available at xxxx
I have been a ChaosHunter (CH) user since it first came out. My interest in CH was to try to learn new relationships about various data series. I was hoping to find a predictive relationship that would be generalizable to various securities in various time periods.
My early work found some promising formulas for individual securities, but I was not successful finding relationships that could be applied to other securities. Recently I tried a different approach with very different results.
Working with a daily SPY chart in NeuroShell Trader, I created three separate predictions, each with a single input taken from my BMJ Software fractal filter indicators. The idea was to find the parameters of my indicators that would be the best predicators of the data series. In other words they would, ideally, be fractal filtered versions of the data series.
Once the predictions found the best parameters for my fractal indicators, I exported the price and volume data from the chart and the fractal indicators with the refined parameters to a .csv file for import into CH.
In the CH input section I selected open, high, low, close, volume, and my three indicators as inputs and chose open as the output. On the optimization tab, I selected evolution strategy and all of the Arithmetic operations and all of the Technical Indicators applied to all of the inputs.
When I optimized the model, the results progressed through some fairly simple arithmetic relationships and settled on a very simple calculation using one of the standard inputs and one of my indicators. The result was a very simple calculation based on a very simple relationship that I would never have thought of on my own. This produced a significant number of trades and a very smooth equity curve.
When I inserted the formula in a NeuroShell chart, the results were very promising. However, I was more interested in using the power of NeuroShell to create a trading strategy that could be applied to multiple symbols. So I decided to create a new trading strategy in NeuroShell based on the relationships identified by CH.
I built a trading strategy in NeuroShell so that the long entry was based on conceptually similar but different inputs than the short entry. This allowed the Trader to optimize all of the parameters, including the parameters of my fractal filter indicators. The results appeared to be more robust and generalizable than the results from the CH model. Based on my testing of the new trading strategy on various symbols, I was able to develop two additional versions of the trading strategy with slight modifications of the inputs. Next I combined the three versions into an ensemble trading strategy.
The results for the ensemble system were very good for both the optimization period and the out-of-sample period, beating the buy and hold by greater than 8 to 1 in the optimization period and greater than 14 to 1 in the out-of-sample period.
I have since tested this trading strategy with multiple symbols from E-minis to commodities to Forex to individual stocks with similar results, in various time frames from daily to weekly to monthly bars as well as intraday.
To try to be sure that the model is working in a fashion that is consistent with the original modeling accomplished by CH, I created 5 charts for 5 of the components of the DJIA – AA, CVX, MRK, UTX, and DIS. I optimized for the period from 1/1/2004 through 2/28/2012 (the end point for the original NeuroShell and CH optimizations) and held the data from 3/1/2012 through 4/7/2012 out of sample. The results beat the buy and hold strategy by a substantial margin for all 5 symbols in both the optimization period and the out-of-sample period.
I have a test system running the five E-minis (DJIA, S&P 500, NASDAQ 100, Russell 2000, and S&P Midcap 400) using NeuroShell automated trading with an InteractiveBrokers simulated account. So far the results seem very promising.
My conclusion at this stage is that CH was able to find a relationship that I would never have dreamed of, a relationship that is repeatable in various time frames with various price patterns, and a relationship that is generalizable to various securities. And I was able to use the powerful features of NeuroShell to preprocess the data for CH and then to improve on and test the relationships that were suggested by CH by creating a sophisticated ensemble trading strategy in NeuroShell.
For more information, contact Dennis Voigt from BMJ Software at dvoigt@bmjsoftware.com. His web site is www.bmjsoftware.com
***********************************************************
V. 20% discount on NeuroShell Trader upgrades and ChaosHunter
We are also offering a 20% discount on NeuroShell Trader upgrades to any Power User version until May 31, 2012. We’re also offering a 20% discount on the purchase of ChaosHunter. Simply enter 20% May newsletter discount in the comments box on the order form. To order the Trader upgrades, go to www.ward.net/60.asp. To order ChaosHunter, go to www.chaoshunter.com.
************************************************************
VI. Office closed on Monday, May 28, for Memorial Day
The Ward Systems Group office will be closed on May 28, 2012, in honor of the Memorial Day holiday and will reopen on Tuesday, May 29.
***********************************************************
VII. One way to stop this newsletter
It is really easy. Just change your email address and don’t tell us.
***********************************************************