Publications | Empirical Inference - Max Planck Institute for Intelligent Systems

This thesis extends the use of kernel learning techniques to specific problems of image classification. Kernel learning is a paradigm in the field of machine learning that generalizes the use of inner products to compute similarities between arbitrary objects. In image classification one aims to separate images based on their visual content. We address two important problems that arise in this context: learning with weak label information and combination of heterogeneous data sources. The contributions we report on are not unique to image classification, and apply to a more general class of problems. We study the problem of learning with label ambiguity in the multiple instance learning framework. We discuss several different image classification scenarios that arise in this context and argue that the standard multiple instance learning requires a more detailed disambiguation. Finally we review kernel learning approaches proposed for this problem and derive a more efficient algorithm to solve them. The multiple kernel learning framework is an approach to automatically select kernel parameters. We extend it to its infinite limit and present an algorithm to solve the resulting problem. This result is then applied in two directions. We show how to learn kernels that adapt to the special structure of images. Finally we compare different ways of combining image features for object classification and present significant improvements compared to previous methods.

PDF [BibTex]

2009

Gehler, PV. Kernel Learning Approaches for Image Classification Biologische Kybernetik, Universität des Saarlandes, Saarbrücken, Germany, October 2009 (phdthesis)

PDF [BibTex]

A PAC-Bayesian Approach to Structure Learning

Seldin, Y.

Biologische Kybernetik, The Hebrew University of Jerusalem, Israel, September 2009 (phdthesis)

PDF [BibTex]

Seldin, Y. A PAC-Bayesian Approach to Structure Learning Biologische Kybernetik, The Hebrew University of Jerusalem, Israel, September 2009 (phdthesis)

PDF [BibTex]

Kernel Methods in Computer Vision:Object Localization, Clustering,and Taxonomy Discovery

Blaschko, MB.

Biologische Kybernetik, Technische Universität Berlin, Berlin, Germany, March 2009 (phdthesis)

PDF PDF [BibTex]

Blaschko, MB. Kernel Methods in Computer Vision:Object Localization, Clustering,and Taxonomy Discovery Biologische Kybernetik, Technische Universität Berlin, Berlin, Germany, March 2009 (phdthesis)

PDF PDF [BibTex]

Motor Control and Learning in Table Tennis

Mülling, K.

Eberhard Karls Universität Tübingen, Gerrmany, 2009 (diplomathesis)

[BibTex]

Mülling, K. Motor Control and Learning in Table Tennis Eberhard Karls Universität Tübingen, Gerrmany, 2009 (diplomathesis)

[BibTex]

Hierarchical Clustering and Density Estimation Based on k-nearest-neighbor graphs

Drewe, P.

Eberhard Karls Universität Tübingen, Germany, 2009 (diplomathesis)

[BibTex]

Drewe, P. Hierarchical Clustering and Density Estimation Based on k-nearest-neighbor graphs Eberhard Karls Universität Tübingen, Germany, 2009 (diplomathesis)

[BibTex]

Learning with Structured Data: Applications to Computer Vision

Nowozin, S.

Technische Universität Berlin, Germany, 2009 (phdthesis)

PDF [BibTex]

Nowozin, S. Learning with Structured Data: Applications to Computer Vision Technische Universität Berlin, Germany, 2009 (phdthesis)

PDF [BibTex]

From Differential Equations to Differential Geometry: Aspects of Regularisation in Machine Learning

Steinke, F.

Universität des Saarlandes, Saarbrücken, Germany, 2009 (phdthesis)

PDF [BibTex]

Steinke, F. From Differential Equations to Differential Geometry: Aspects of Regularisation in Machine Learning Universität des Saarlandes, Saarbrücken, Germany, 2009 (phdthesis)

PDF [BibTex]

2005

Extension to Kernel Dependency Estimation with Applications to Robotics

BakIr, G.

Biologische Kybernetik, Technische Universität Berlin, Berlin, November 2005 (phdthesis)

Abstract

Kernel Dependency Estimation(KDE) is a novel technique which was designed to learn mappings between sets without making assumptions on the type of the involved input and output data. It learns the mapping in two stages. In a first step, it tries to estimate coordinates of a feature space representation of elements of the set by solving a high dimensional multivariate regression problem in feature space. Following this, it tries to reconstruct the original representation given the estimated coordinates. This thesis introduces various algorithmic extensions to both stages in KDE. One of the contributions of this thesis is to propose a novel linear regression algorithm that explores low-dimensional subspaces during learning. Furthermore various existing strategies for reconstructing patterns from feature maps involved in KDE are discussed and novel pre-image techniques are introduced. In particular, pre-image techniques for data-types that are of discrete nature such as graphs and strings are investigated. KDE is then explored in the context of robot pose imitation where the input is a an image with a human operator and the output is the robot articulated variables. Thus, using KDE, robot pose imitation is formulated as a regression problem.

PDF PDF [BibTex]

2005

BakIr, G. Extension to Kernel Dependency Estimation with Applications to Robotics Biologische Kybernetik, Technische Universität Berlin, Berlin, November 2005 (phdthesis)

PDF PDF [BibTex]

Geometrical aspects of statistical learning theory

Hein, M.

Biologische Kybernetik, Darmstadt, Darmstadt, November 2005 (phdthesis)

PDF [BibTex]

Hein, M. Geometrical aspects of statistical learning theory Biologische Kybernetik, Darmstadt, Darmstadt, November 2005 (phdthesis)

PDF [BibTex]

Implicit Surfaces For Modelling Human Heads

Steinke, F.

Biologische Kybernetik, Eberhard-Karls-Universität, Tübingen, September 2005 (diplomathesis)

[BibTex]

Steinke, F. Implicit Surfaces For Modelling Human Heads Biologische Kybernetik, Eberhard-Karls-Universität, Tübingen, September 2005 (diplomathesis)

[BibTex]

Machine Learning Methods for Brain-Computer Interdaces

Lal, TN.

Biologische Kybernetik, University of Darmstadt, September 2005 (phdthesis)

Web [BibTex]

Lal, TN. Machine Learning Methods for Brain-Computer Interdaces Biologische Kybernetik, University of Darmstadt, September 2005 (phdthesis)

Web [BibTex]

Liver Perfusion using Level Set Methods

Nowozin, S.

Biologische Kybernetik, Shanghai JiaoTong University, Shanghai, China, July 2005 (diplomathesis)

PDF [BibTex]

Nowozin, S. Liver Perfusion using Level Set Methods Biologische Kybernetik, Shanghai JiaoTong University, Shanghai, China, July 2005 (diplomathesis)

PDF [BibTex]

Discriminative Methods for Label Sequence Learning

Altun, Y.

Brown University, Providence, RI, USA, May 2005 (phdthesis)

PDF [BibTex]

Altun, Y. Discriminative Methods for Label Sequence Learning Brown University, Providence, RI, USA, May 2005 (phdthesis)

PDF [BibTex]

Efficient Adaptive Sampling of the Psychometric Function by Maximizing Information Gain

Tanner, TG.

Biologische Kybernetik, Eberhard-Karls University Tübingen, Tübingen, Germany, May 2005 (diplomathesis)

Abstract

A common task in psychophysics is to measure the psychometric function. A psychometric function can be described by its shape and four parameters: offset or threshold, slope or width, false alarm rate or chance level and miss or lapse rate. Depending on the parameters of interest some points on the psychometric function may be more informative than others. Adaptive methods attempt to place trials on the most informative points based on the data collected in previous trials. A new Bayesian adaptive psychometric method placing trials by minimising the expected entropy of the posterior probabilty dis- tribution over a set of possible stimuli is introduced. The method is more flexible, faster and at least as efficient as the established method (Kontsevich and Tyler, 1999). Comparably accurate (2dB) threshold and slope estimates can be obtained after about 30 and 500 trials, respectively. By using a dynamic termination criterion the efficiency can be further improved. The method can be applied to all experimental designs including yes/no designs and allows acquisition of any set of free parameters. By weighting the importance of parameters one can include nuisance parameters and adjust the relative expected errors. Use of nuisance parameters may lead to more accurate estimates than assuming a guessed fixed value. Block designs are supported and do not harm the performance if a sufficient number of trials are performed. The method was evaluated by computer simulations in which the role of parametric assumptions, its robustness, the quality of different point estimates, the effect of dynamic termination criteria and many other settings were investigated.

[BibTex]

Tanner, TG. Efficient Adaptive Sampling of the Psychometric Function by Maximizing Information Gain Biologische Kybernetik, Eberhard-Karls University Tübingen, Tübingen, Germany, May 2005 (diplomathesis)

[BibTex]

Support Vector Classification of Images with Local Features

Blaschko, MB.

Biologische Kybernetik, University of Massachusetts, Amherst, May 2005 (diplomathesis)

[BibTex]

Blaschko, MB. Support Vector Classification of Images with Local Features Biologische Kybernetik, University of Massachusetts, Amherst, May 2005 (diplomathesis)

[BibTex]

Efficient Pattern Selection for Support Vector Classifiers and its CRM Application

Shin, H.

Biologische Kybernetik, Seoul National University, Seoul, Korea, February 2005 (phdthesis)

PDF [BibTex]

Shin, H. Efficient Pattern Selection for Support Vector Classifiers and its CRM Application Biologische Kybernetik, Seoul National University, Seoul, Korea, February 2005 (phdthesis)

PDF [BibTex]

Kernels: Regularization and Optimization

Ong, CS.

Biologische Kybernetik, The Australian National University, Canberra, Australia, 2005 (phdthesis)

PDF GZIP [BibTex]

Ong, CS. Kernels: Regularization and Optimization Biologische Kybernetik, The Australian National University, Canberra, Australia, 2005 (phdthesis)

PDF GZIP [BibTex]

2002

Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond

Schölkopf, B., Smola, A.

pages: 644, Adaptive Computation and Machine Learning, MIT Press, Cambridge, MA, USA, December 2002, Parts of this book, including an introduction to kernel methods, can be downloaded here. (book)

Abstract

In the 1990s, a new type of learning algorithm was developed, based on results from statistical learning theory: the Support Vector Machine (SVM). This gave rise to a new class of theoretically elegant learning machines that use a central concept of SVMs-kernelsfor a number of learning tasks. Kernel machines provide a modular framework that can be adapted to different tasks and domains by the choice of the kernel function and the base algorithm. They are replacing neural networks in a variety of fields, including engineering, information retrieval, and bioinformatics. Learning with Kernels provides an introduction to SVMs and related kernel methods. Although the book begins with the basics, it also includes the latest research. It provides all of the concepts necessary to enable a reader equipped with some basic mathematical knowledge to enter the world of machine learning using theoretically well-founded yet easy-to-use kernel algorithms and to understand and apply the powerful algorithms that have been developed over the last few years.

Web [BibTex]

2002

Schölkopf, B., Smola, A. Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond pages: 644, Adaptive Computation and Machine Learning, MIT Press, Cambridge, MA, USA, December 2002, Parts of this book, including an introduction to kernel methods, can be downloaded here. (book)

Web [BibTex]

Nonlinear Multivariate Analysis with Geodesic Kernels

Kuss, M.

Biologische Kybernetik, Technische Universität Berlin, February 2002 (diplomathesis)

GZIP [BibTex]

Kuss, M. Nonlinear Multivariate Analysis with Geodesic Kernels Biologische Kybernetik, Technische Universität Berlin, February 2002 (diplomathesis)

GZIP [BibTex]

Concentration Inequalities and Empirical Processes Theory Applied to the Analysis of Learning Algorithms

Bousquet, O.

Biologische Kybernetik, Ecole Polytechnique, 2002 (phdthesis) Accepted

Abstract

New classification algorithms based on the notion of 'margin' (e.g. Support Vector Machines, Boosting) have recently been developed. The goal of this thesis is to better understand how they work, via a study of their theoretical performance. In order to do this, a general framework for real-valued classification is proposed. In this framework, it appears that the natural tools to use are Concentration Inequalities and Empirical Processes Theory. Thanks to an adaptation of these tools, a new measure of the size of a class of functions is introduced, which can be computed from the data. This allows, on the one hand, to better understand the role of eigenvalues of the kernel matrix in Support Vector Machines, and on the other hand, to obtain empirical model selection criteria.

PostScript [BibTex]

Bousquet, O. Concentration Inequalities and Empirical Processes Theory Applied to the Analysis of Learning Algorithms Biologische Kybernetik, Ecole Polytechnique, 2002 (phdthesis) Accepted

PostScript [BibTex]

Support Vector Machines: Induction Principle, Adaptive Tuning and Prior Knowledge

Chapelle, O.

Biologische Kybernetik, 2002 (phdthesis)

Abstract

This thesis presents a theoretical and practical study of Support Vector Machines (SVM) and related learning algorithms. In a first part, we introduce a new induction principle from which SVMs can be derived, but some new algorithms are also presented in this framework. In a second part, after studying how to estimate the generalization error of an SVM, we suggest to choose the kernel parameters of an SVM by minimizing this estimate. Several applications such as feature selection are presented. Finally the third part deals with the incoporation of prior knowledge in a learning algorithm and more specifically, we studied the case of known invariant transormations and the use of unlabeled data.

GZIP [BibTex]