NOTE: In NeuroShell Trader, open the chart named “Example 9 – Optimization with Evaluation” which is the basis for following example:

In this example we examine a modeling problem common to all model building with past data.

Any time you optimize there is a possibility that you may “over-fit”, which means that you build a model that worked so well in the past that it fit the past noise too, so that it probably will not work well into the future. (Editors note: some people use the word “curve fitting” to describe the over-fitting phenomenon, but that is a misnomer because all modeling from past data is curve fitting – it is over curve fitting, i.e. or over-fitting, that is the problem).

Over-fitting can occur even if you do not optimize, because in fact when you backtest different strategies to see which works the best, you are in fact optimizing manually! However, machine optimizing increases the odds of over-fitting because it is so much more efficient. The possibility of over-fitting is reduced by not optimizing, optimizing over plenty of historical data, and/or optimizing as few parameters as possible.

One way that has been traditionally used to test if over-fitting has occurred is simply to see how the model holds up on new data. There are a couple of ways that can be done:

Of course you could simply watch your model for several weeks or months and “paper trade” it in the traditional sense to see if you would have made money. Few of us want to take the time to do that, however.

Another way to do this in NeuroShell is to use the check box called “Start trading before last chart date” in the Dates tab. That will allow you to select how much time to “hold out” of the optimization for evaluation of the model after it is optimized. Optimization will take place on an earlier period of the chart, and the final backtesting (evaluation) using the model will take place immediately after that. The disadvantage of this “hold out” approach is that even if you are satisfied the model held up during evaluation, you essentially have an old model, one that was not built on the latest data (although it was evaluated on the latest data).

In order to use “Start trading before last chart date”, you must turn on the option that enables it from the menu Tools -> Options -> Advanced -> Data Interface Settings. It is called “Allow real trading to begin before last chart date”.

In the chart of this example we have continued the crossover example, but we went back to optimizing parameters in the short and long rules independently. Recall that in the last example we optimized over all the data that was available at the time. In this example, we optimize over all data EXCEPT the data starting in 2010. Then we allowed the backtest on data starting with 2010. You should examine the chart to see the two separate periods, and then examine the Dates tab in the “Modify Trading Strategy Parameters” button to see how we accomplished the separation of the optimization and the evaluation periods.

Now note that the trading rules (averaging periods) found prior to 2010 did not hold up well after the market started getting volatile in March and April 2010. That is one problem with the typical crossovers many traders use.

At the end of the optimization period, just before the bars change to the green trading period, you will see a close-out of the long position. There is a X fill point on the chart. NeuroShell performs this close-out at the end of a period (even though the strategy did not generate an exit signal) so it can compute separate statistics for each period. The equity curve also starts over on the first bar of the trading period.