Many of you are tempted to do what we’ve seen even seasoned modeling experts do. You train a net and then apply it to the out-of-sample data (production set in NeuroShell 2, evaluation data in NeuroShell Trader, verification data on NeuroShell Predictor and Classifier). It doesn’t work very well, so you change something in your network, and try again. That doesn’t work, so you try something else, etc. We’ve all done that, right?

Well, it’s very bad practice. If you make multiple training runs until you get something that works best on the out-of-sample data, you have just rendered the out-of-sample data set as in-sample, and the net you finally wind up with has to be tested once again (but only once) on NEW out-of-sample data before you can trust it! Why? Because you’ve essentially created a solution specifically for the out-of-sample set.

We see this most often with NeuroShell 2 users, because the older algorithms in NeuroShell 2 have so many parameters to “tweak”.

Here’s a true story. We saw a publicly traded company go out of business after they tried that with FDA on a medical application. They were supposed to build their net on data from one set of patients, and then when they were all done, apply the net to the out-of-sample patients ONCE and deliver the results to FDA. Instead, they kept trying different nets on the out-of-sample patients. The FDA nailed them for “unsealing” the out-of-sample patients (and another data issue), and the FDA was right to do so. The company was forced to do a completely new trial with new patient data. Wall Street reacted negatively to the bad news, and the stock price tumbled. The company was unable to obtain funding for the new trial, and they folded.

So what should you do? There’s no really great answer, but here are some suggestions:

1. NeuroShell 2 users should move up to the NeuroShell Predictor, Classifier, or Trader where there are not so many parameters that have to be tweaked to get a good net. NeuroShell 2 is mostly a tool for academics. The newer tools pretty much give you the best net the first time, although there are always issues of what inputs to use. The genetic method in the Predictor and Classifier, and the GA in the Trader Professional largely solves that problem, though.

2. If you have plenty of data, you can hold out two out-of-sample sets; one to experiment with, and one that you use only once at the end.

3. If you don’t have plenty of data, use the genetic method in the Predictor and Classifier, because it does a “one hold out” even while training. The fit isn’t as good on training as other nets, so most people erroneously reject these nets. However, generalization is usually better, meaning they do better on out-of-sample data.

4. In the NeuroShell Trader Professional or DayTrader Professional, let the optimizer do the playing for you, and then just try once (ok, maybe twice) on the out-of-sample data. If that doesn’t work, move on to a new issue or at least different size training sets and different inputs.

5. In the NeuroShell Trader Professional or DayTrader Professional, use the Paper Trading feature new in release 5.0. You can optionally use a real out-of-sample set with it (called Trading).