#3 is actually wrong. The results of Recht et al. do not show that people are performing validation on the test set. If this were true, one would expect a poor correlation between accuracy on the original CIFAR-10 test set and the new test set, whereas the authors observe an extremely high correlation. The results actually indicate that attempting to follow the same dataset collection procedures as the creators of CIFAR-10 results in a dataset that is slightly harder than the original dataset (at least for models trained on the original dataset). The follow-up paper (http://people.csail.mit.edu/ludwigs/papers/imagenet.pdf) makes this point explicitly. The fact that the relative ordering of models is preserved on the new dataset suggests that the creators of the models didn't cheat, or at least didn't cheat enough to invalidate CIFAR-10 test set performance as an evaluation metric.
> The results of Recht et al. do not show that people are performing validation on the test set.
As I read #3, the hypothesis doesn't depend on individual researchers acting unethically by validating against the test set. Instead, I read it as an analogy to significance bias in other sciences: machine learning models that don't perform as well on the validation set simply aren't published, so the field as a whole over-fits, as if validation is performed on the test set.
In the paper you link, the authors themselves do explicitly note some test-set shenanigans:
> But this assumption [that the models are independent of the test set] is undermined by the common practice of tuning model hyperparameters directly on the test set, which introduces dependencies between the model $\ˆf$ and the test set S. In the extreme case, this can be seen as training directly on the test set
The authors made various claims or implications not backed up by their experiment.
However, it absolutely does not answer the question that is asked in the title of the paper, and the process they use is incapable of answering that question.
If you go back and read the original CIFAR10 paper, you'll see that the process they carefully went through meant that they curated the most suitable images for each category. By definition, what's left over (which is what the Recht et al paper chose from) is less good images, which are of course therefore harder to classify.
All the experiment measures is how good they are at matching the distribution of the original dataset. The answer, they discovered, is: not very.
It is not people are performing validation on the test set, that would be straightly cheating.
What #3 has described is a totally normal workflow:
1. Conceptualize a new model idea
2. Implementing and training is totally legal without involving test set.
3. However, once finished, the model is evaluated on test set, the performance of which will decide whether this idea is worthy of publishing or not. If not, they go to the first step and repeat.
Such loop essentially makes the test, the actual validation set, if you think human as his/her own optimizer, and he/she takes a look at the test set periodically, and decide whether to pursue the current idea or not. Sounds like early stopping, isn't it?
Remember back in 2015, there is a debacle from Baidu, where a researcher had fabricated multiple accounts to run unlimited tests against ImageNet's own reserved test set, which the competition straightly forbade.
If a test set is a 'true' test set, then it should work like a test in a real world: be kept secret before revealing to the public, once evaluated, the same problems/examples shall never appear in the later tests ever. But such approach would not be accepted because the cost is simply too high.
The whole thing is very similar to the replication crisis, which is caused by p being a random variable and sometimes hitting below 0.05 for no reason. Similarly, any training or test set validation percentage is also a random variable. Even a single researcher trains only a single model, the field is very packed which leads to many, many models being created. As test set accuracy is a random variable, some of the models will hit above the threshold and they are the models that can be published. Otherwise the researcher has no results to report. The extra step that eliminates negative results is the mechanism that causes the indirect validation on the test set. Yes, the results are correlated but they also provide inflated results, which is shown by all of the new test accuracy points lying wildly outside the prediction interval.
I don't think machine learning suffers from the same kind of p-value-driven replication crisis as other fields. It is true that people don't generally perform proper statistics to compare machine learning models [1], but ML research has two things going for it that other scientific fields do not. First, comparing machine learning models on the same test set corresponds to a within-subjects analysis, generally with tens of thousands of subjects, so the noise level is low. Second, because ML researchers don't generally perform hypothesis tests, they care solely about effect size and not about significance. If my model gets all the same examples right as the previous state-of-the-art, plus 10 more, then my model is statistically significantly better, but on a test set of 10,000 examples this corresponds to a 0.1% accuracy improvement, which is generally not big enough to publish. By not doing hypothesis tests, ML researchers actually tend to be more conservative than their p-value-driven counterparts in other fields.
In the Recht et al. study, the reason the new test accuracy is wildly outside of a binomial confidence interval around the original test set accuracy is that the distribution is different. The CI only applies to data drawn from the same distribution.
ML research still suffers from replication issues; such is the nature of the scientific incentive structure. However, these issues generally come in the form of poorly tuned baselines, buggy code, and claims with insufficient experimental/theoretical justification. Outside of some isolated cases, publication bias and cheating at hyperparameter tuning do not seem to be major factors.
----
[1] Statistically speaking, to compare two models on the same dataset, one does not care about the accuracy numbers but instead about the number of examples model A gets right that model B does not and vice versa; see McNemar's test.
The problem is that outside of the machine learning community people don't hear "within-subjects analysis" they hear (and are told) "better than human performance". Within the community I think you are right; people are working on a shared set of assumptions and have the same expectations about performance in the real world (that the results will not transfer without massive negative deltas), but that is definitely not what 10000's of web developers downloading scikit-learn or tensorflow believe.
