Semiparametric regression using variational approximations

The New South Wales branch held its first meeting of the year on March 28 at UTS and was fortunate to have Dr Francis Hui agree to present at short notice. Dr Hui is a lecturer in statistics at ANU, and is undoubtedly one of Australia’s leading young statisticians. Introduced as being “animated, entertaining and technically precise”, he lived up to the billing as he provided a high-energy overview of his recent research.

Francis began with an introduction to generalized additive models (GAMs), and described the two main methods for estimation. Maximum penalized likelihood — the approach taken by the excellent mgcv package — is very fast but biased and can be highly unstable in small samples. Furthermore, it requires an external procedure such as generalized cross-validation in order to choose the parameter that controls the amount of smoothing. On the other hand, a mixed model (GAMM) approach is stable and provides a framework for inference as we are maximizing a proper likelihood, as well as allowing for simultaneous estimation of the smoothing parameter. However, for non-normal responses the likelihood involves an intractable integral, and estimation is often slow.

Ideally we would like the best of both worlds, and to this end, Francis introduced the concept of variational approximations (VA). Under this method, we replace an intractable integral by a closed-form lower bound, which becomes the objective function for our maximisation. When applied to GAMs, a VA approach means that we can still use the mixed models framework but are now trying to maximise something that looks like a penalized likelihood, potentially providing the “match made in statistical heaven” that we are looking for.

VAGAMs incorporate selection of the smoothing parameter and suggest approaches that can be taken to perform inference on both the smooth and parametric components. Asymptotically, variational estimates are consistent and normally distributed, and the efficiency and rate of convergence are the same as in standard maximum likelihood estimation. In simulations, Dr Hui showed that VAGAM performed very well in comparison to penalized likelihood while being substantially faster than GAMM fitting.

Dr Hui is looking for post-docs and PhD students interested in dimension reduction, variable selection, fast estimation and inference for complex correlated data, and spending some time in lovely little Canberra. Please contact for more details regarding projects and funding opportunities!

Mark Donoghoe


Get the latest posts delivered to your mailbox:

Show Buttons
Hide Buttons