The following explains the release 5.x in-sample and out-of-sample features in wording different than either the help files or the examples.

Both the Prediction Wizard and the Trading Wizard now work the same way. There are three contiguous periods of time you can select from:

1. Training/optimization – associated with gray on the chart, and the time period in the past during which training and optimization takes place.

2. Paper Trading – associated with orange on the chart, and the time period in the past following period 1 training/optimization during which each model found during the training/optimization is separately backtested to see how it does on future out-of-sample data. The model that works best on this period 2 is kept, even though the objective of optimization/training is to do best during the training/optimization period 1. In other words, we are not keeping the best model for period 1. We are keeping the best model for period 2 which was noticed during the evolution of the best model for period 1. That will not necessarily be the same as the best model for period 1. It will also probably not be as good as the best model possible for period 2 if we had actually optimized over period 2.

3. Trading/evaluation – associated with green on the chart, and the time period following paper trading period 2 (if there is a period 2, or after period 1 if there is no period 2) during which the model is separately backtested for a true out-of-sample experience.

If you train and optimize in period 1 you do not have to have either of the out-of-sample periods 2 or 3. You can have neither, either, or both following period 1. We’ll explain:

I. Neither – you train/optimize and do no further testing at all.

II. Either

a. If you have only a paper trading period, you go forward with the model optimized over period 1 but found best in the out-of-sample paper trading period 2. This is equivalent to trying many models, then testing each one out-of-sample to see which one worked best in later out-of-sample. We have thus automated the process almost everyone follows anyway.

b. If you have only a trading/evaluation period, you go forward with the model that worked best during period 1 training/optimization, but you see how it did on the contiguous out-of-sample period. There is no paper trading period 2 in this case, and no bars are reserved for it.

III. Both – in this case, you keep the model that worked best during period 2 out-of-sample paper trading, but in the end it is still separately backtested over another out-of-sample period 3, under the assumption that with all the out-of-sample testing in period 2 you may need yet a final out-of-sample test to maintain purity (if you think such purity is essential, which we believe is true if you are testing new medicines, but arguable in financial markets).