Probability Theory is an integral part of machine learning as a discipline, and continues to play a major role in the development of the ﬁeld. The modularity of probabilistic models has repeatedly allowed for them to be extended and developed into general frameworks

for large classes of problems, like regression and density estimation. In many areas of statistics and machine learning, probabilistic formulations were not the ﬁrst to the table, but still helped improve understanding of implicit assumptions of the established statistical algorithms, helped with theoretical analysis, and lit the way to algorithmic extensions. Over the years, members of the department of empirical inference have contributed substantially to the ﬁeld of probabilistic learning.

Probability Theory is an integral part of machine learning as a discipline, and continues to play a major role in the development of the field. The modularity of probabilistic models has repeatedly allowed for them to be extended and developed into general frameworks for large classes of problems, like regression and density estimation. In many areas of statistics and machine learning, probabilistic formulations were not the first to the table, but still helped improve understanding of implicit assumptions of the established statistical algorithms, helped with theoretical analysis, and lit the way to algorithmic extensions. Over the years, members of the department of empirical inference have contributed substantially to the field of probabilistic learning.

**Gaussian Process Regression**: The textbook on Gaussian processes for machine learning, written by Carl Rasmussen in collaboration with Christopher Williams\footnote{Rasmussen \& Williams, \emph{Gaussian Processes for Machine Learning}, MIT Press, 2006} has played a major role in the development of the Gaussian process framework into a workhorse of machine learning. In 2009, the book was awarded the deGroot Prize for Best Text on Bayesian Inference. Nickisch and Rasmussen pubLink{5305} wrote a comprehensive overview of Gaussian process classification. Rasmussen and Nickisch pubLink{6779} created the GPML toolbox, by far the most comprehensive package for Gaussian processes; originally, this was conceived as the code for the book, but by now it greatly exceeds models and algorithms discussed in the textbook, and is still being actively developed. Duvenaud, Nickisch and Rasmussen pubLink{DuvenaudNR2012} have provided a recent contribution on structure in Gaussian process models using classical concepts of additivity. Nickisch pubLink{6939} developed a toolbox for generalized linear models, providing scalable, efficient routines for inference in sparse linear, logistic, and abstract classification models.

**Approximate Inference Methods**: The computational complexity of probabilistic formulations demands good numeric and algebraic approximation methods. Consequently, probabilistic models and approximation methods have developed alongside each other. Among the contributions to this area from members of the department, Habeck pubLink{Habeck2012_2,Habeck2012_3} introduced a framework for the estimation of free energies and densities of state from equilibrium Monte Carlo simulations, offering a more robust and flexible alternative to thermodynamic integration. Seeger and Nickisch pubLink{SeegerN2011} developed a highly efficient, and provably convergent algorithmic improvement to the Expectation Propagation framework. Hennig et al. pubLink{HennigSHG2012} contributed a lightweight, and numerically stable approximation for inference in latent Dirichlet models, allowing for efficient latent nonparametric dependency modeling between discrete distributions.

**Probabilistic Numerical Methods**: Problems of numerical mathematics, like sampling, quadrature and optimization can be interpreted as inference problems: Their goal is to infer unknown properties of a function, like the value of a definite integral or the locations of extrema, from evaluations of the function at various locations. Even if the function is analytically known and can be evaluated at any point, these quantities of interest are truly unknown, otherwise it would not be necessary to call a numerical algorithm. This view of numerics as inference is not new, but has recently attracted renewed attention, partly due to work carried out in the department. Hennig \& Schuler pubLink{HennigS2012} used a probabilistic formulation of the global optimization problem to develop a sample-efficient experimental design algorithm that chooses evaluation locations based on expected information gain about the optimum. More recently, Hennig \& Kiefel pubLink{HennigK2012} performed a probabilistic analysis of classic quasi-Newton methods, like the widely-used BFGS algorithm, to identify algebraic prior assumptions made within these algorithms, and constructed a nonparametric extension of the quasi-Newton framework. Members of the department are helping to shape the discussion in this area, for example by co-organising a NIPS workshop on probabilistic numerics, in December 2012.

9 results

**Bayesian estimation of free energies from equilibrium simulations**
*Physical Review Letters*, 109(10):5, September 2012 (article)

**Entropy Search for Information-Efficient Global Optimization**
*Journal of Machine Learning Research*, 13, pages: 1809-1837, -, June 2012 (article)

**glm-ie: The Generalised Linear Models Inference and Estimation Toolbox**
*Journal of Machine Learning Research*, 13, pages: 1699-1703, May 2012 (article)

**Evaluation of marginal likelihoods via the density of states**
In *Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics (AISTATS 2012)
*, 22, pages: 486-494, (Editors: N Lawrence and M Girolami), JMLR: W&CP 22, AISTATS, 2012 (inproceedings)

**Kernel Topic Models**
In *Fifteenth International Conference on Artificial Intelligence and Statistics*, 22, pages: 511-519, JMLR Proceedings, (Editors: Lawrence, N. D. and Girolami, M.), JMLR.org, AISTATS , 2012 (inproceedings)

**Additive Gaussian Processes**
In *Advances in Neural Information Processing Systems 24*, pages: 226-234, (Editors: J Shawe-Taylor and RS Zemel and P Bartlett and F Pereira and KQ Weinberger), Twenty-Fifth Annual Conference on Neural Information Processing Systems (NIPS), 2011 (inproceedings)

**Fast Convergent Algorithms for Expectation Propagation Approximate Bayesian Inference **
In *JMLR Workshop and Conference Proceedings Volume 15: AISTATS 2011*, pages: 652-660, (Editors: Gordon, G. , D. Dunson, M. Dudík ), MIT Press, Cambridge, MA, USA, 14th International Conference on Artificial Intelligence and Statistics, April 2011 (inproceedings)

**Gaussian Processes for Machine Learning (GPML) Toolbox**
*Journal of Machine Learning Research*, 11, pages: 3011-3015, November 2010 (article)

**Approximations for Binary Gaussian Process Classification**
*Journal of Machine Learning Research*, 9, pages: 2035-2078, October 2008 (article)