Patrick,

I had not heard Neyman’s quote about induction. I guess he and Popper would not have had a beer together!

I think my main problem is that CV, in the context I have described it, does not answer the question I care about. I think I can see what its use is now, see my response to John.

Interestingly, AI warned me off looic – not that I was considering it. It claimed that it increases the variance of the predictive accuracy estimate.

Thanks you all  for your interesting replies. I will respond in order, in separate posts.

Andrew,

I think that many questions can be answered from a frequentist perspective. What is the coverage of this confidence interval? What is the standard error of this estimate? There are Bayesian answers to these questions, but you can get a sensible and relevant answer without the need to invoke priors.

It seems to me that CV does answer a frequentist question, but it is not a question I want to ask. I want to know the out-of-sample predictive accuracy of this model, because I am going to use this model in future.

A proper conditional frequentist answer would be conditional on the model but no other aspects of the data. If you do not care about model mis=specification then bootstrap can give a good approximation to this. But we should care about model mis-specification. Bayesians do this automatically, because they condition on the entire data, which includes the selected model that it resulted in.

So, if you want to know out-of-sample accuracy, I think you need some out-of-sample data. Just no way out of it.

The benefits of k-fold cross-validation are surely 1) that it can be used to account for the selection effects of the strategy used to select the model and 2) that it uses all the data.  Obviously, it does not account for any difference between the data used to train the data and the data to which it will be applied.  For that a training/test approach is needed, or if one is being very careful, a training/validate/test approach.  These sorts of approaches are crucial if AI is used in any substantial way, e.g., to select a model.

I have been interrogating AI about k-fold cross-validation for a course I am writing. I thought I understood it before, but I understand it better now. AI is really brilliant for this kind of thing. It is like talking to an enthusiastic RA who is smart and knowledgeable, makes some mistakes, but always sucks up to you because you hold the research grant.  ;)

Anyway, it seems to me that the cross-validation is b-s, at least in any application I can think of. So I am intending not to teach it, or at least give it a couple of slides and say not to bother.

Hear me out.

CV gives you the accuracy of the modelling strategy, not the model you actually use. It estimates accuracy averaged over different possible models from different random subsets of the data. And then at the end, you apply the modelling strategy to all the data and say that the CV estimate described it. But it doesn’t describe the model; it describes the modelling strategy.

Now I am a pretty hard core frequentist. But in this case, I want to estimate future prediction accuracy conditional on all the (training) data that led to the model. I do not want to average over models that might have been! 

It's like estimating my life outcome averaging over different decisions I could have made over the last 68 years.

That’s why Kaggle just uses partition – leaving some data out and judging the winner based on this test data.

Help me out here! Have I completely missed the point? Am I becoming a closet Bayesian in my old age?!

It could happen.!Fred Hoyle became a catholic after all....