Publications | Empirical Inference - Max Planck Institute for Intelligent Systems

614 results (View BibTeX file of all listed publications)

2006

Gaussian Process Models for Robust Regression, Classification, and Reinforcement Learning

Biologische Kybernetik, Technische Universität Darmstadt, Darmstadt, Germany, March 2006, passed with distinction, published online (phdthesis)

PDF [BibTex]

2006

Kuss, M. Gaussian Process Models for Robust Regression, Classification, and Reinforcement Learning Biologische Kybernetik, Technische Universität Darmstadt, Darmstadt, Germany, March 2006, passed with distinction, published online (phdthesis)

PDF [BibTex]

Classification of Natural Scenes: Critical Features Revisited

Drewes, J., Wichmann, F., Gegenfurtner, K.

9, pages: 92, 9th T{\"u}bingen Perception Conference (TWK), March 2006 (poster)

Abstract

Human observers are capable of detecting animals within novel natural scenes with remarkable speed and accuracy. Despite the seeming complexity of such decisions it has been hypothesized that a simple global image feature, the relative abundance of high spatial frequencies at certain orientations, could underly such fast image classification [1]. We successfully used linear discriminant analysis to classify a set of 11.000 images into animal and non-animal images based on their individual amplitude spectra only [2]. We proceeded to sort the images based on the performance of our classifier, retaining only the best and worst classified 400 images ("best animals", "best distractors" and "worst animals", "worst distractors"). We used a Go/No-go paradigm to evaluate human performance on this subset of our images. Both reaction time and proportion of correctly classified images showed a significant effect of classification difficulty. Images more easily classified by our algorithm were also classified faster and better by humans, as predicted by the Torralba & Oliva hypothesis. We then equated the amplitude spectra of the 400 images, which, by design, reduced algorithmic performance to chance whereas human performance was only slightly reduced [3]. Most importantly, the same images as before were still classified better and faster, suggesting that even in the original condition features other than specifics of the amplitude spectrum made particular images easy to classify, clearly at odds with the Torralba & Oliva hypothesis.

Web [BibTex]

Drewes, J., Wichmann, F., Gegenfurtner, K. Classification of Natural Scenes: Critical Features Revisited 9, pages: 92, 9th T{\"u}bingen Perception Conference (TWK), March 2006 (poster)

Web [BibTex]

Machine Learning Methods For Estimating Operator Equations

Steinke, F., Schölkopf, B.

In Proceedings of the 14th IFAC Symposium on System Identification (SYSID 2006), pages: 6, (Editors: B Ninness and H Hjalmarsson), Elsevier, Oxford, United Kingdom, 14th IFAC Symposium on System Identification (SYSID), March 2006 (inproceedings)

Abstract

We consider the problem of fitting a linear operator induced equation to point sampled data. In order to do so we systematically exploit the duality between minimizing a regularization functional derived from an operator and kernel regression methods. Standard machine learning model selection algorithms can then be interpreted as a search of the equation best fitting given data points. For many kernels this operator induced equation is a linear differential equation. Thus, we link a continuous-time system identification task with common machine learning methods. The presented link opens up a wide variety of methods to be applied to this system identification problem. In a series of experiments we demonstrate an example algorithm working on non-uniformly spaced data, giving special focus to the problem of identifying one system from multiple data recordings.

PDF Web [BibTex]

Steinke, F., Schölkopf, B. Machine Learning Methods For Estimating Operator Equations In Proceedings of the 14th IFAC Symposium on System Identification (SYSID 2006), pages: 6, (Editors: B Ninness and H Hjalmarsson), Elsevier, Oxford, United Kingdom, 14th IFAC Symposium on System Identification (SYSID), March 2006 (inproceedings)

PDF Web [BibTex]

Data mining problems and solutions for response modeling in CRM

Cho, S., Shin, H., Yu, E., Ha, K., MacLachlan, D.

Entrue Journal of Information Technology, 5(1):55-64, March 2006 (article)

Abstract

We present three data mining problems that are often encountered in building a response model. They are robust modeling, variable selection and data selection. Respective algorithmic solutions are given. They are bagging based ensemble, genetic algorithm based wrapper approach and nearest neighbor-based data selection in that order. A real world data set from Direct Marketing Educational Foundation, or DMEF4, is used to show their effectiveness. Proposed methods were found to solve the problems in a practical way.

PDF [BibTex]

Cho, S., Shin, H., Yu, E., Ha, K., MacLachlan, D. Data mining problems and solutions for response modeling in CRM Entrue Journal of Information Technology, 5(1):55-64, March 2006 (article)

PDF [BibTex]

Factorial Coding of Natural Images: How Effective are Linear Models in Removing Higher-Order Dependencies?

Bethge, M.

9, pages: 90, 9th T{\"u}bingen Perception Conference (TWK), March 2006 (poster)

Abstract

The performance of unsupervised learning models for natural images is evaluated quantitatively by means of information theory. We estimate the gain in statistical independence (the multi-information reduction) achieved with independent component analysis (ICA), principal component analysis (PCA), zero-phase whitening, and predictive coding. Predictive coding is translated into the transform coding framework, where it can be characterized by the constraint of a triangular filter matrix. A randomly sampled whitening basis and the Haar wavelet are included into the comparison as well. The comparison of all these methods is carried out for different patch sizes, ranging from 2x2 to 16x16 pixels. In spite of large differences in the shape of the basis functions, we find only small differences in the multi-information between all decorrelation transforms (5% or less) for all patch sizes. Among the second-order methods, PCA is optimal for small patch sizes and predictive coding performs best for large patch sizes. The extra gain achieved with ICA is always less than 2%. In conclusion, the `edge filters‘ found with ICA lead only to a surprisingly small improvement in terms of its actual objective.

Web [BibTex]

Bethge, M. Factorial Coding of Natural Images: How Effective are Linear Models in Removing Higher-Order Dependencies? 9, pages: 90, 9th T{\"u}bingen Perception Conference (TWK), March 2006 (poster)

Web [BibTex]

Implicit Volterra and Wiener Series for Higher-Order Image Analysis

Franz, M., Schölkopf, B.

In Advances in Data Analysis: Proceedings of the 30th Annual Conference of The Gesellschaft für Klassifikation, 30, pages: 1, March 2006 (inproceedings)

Abstract

The computation of classical higher-order statistics such as higher-order moments or spectra is difficult for images due to the huge number of terms to be estimated and interpreted. We propose an alternative approach in which multiplicative pixel interactions are described by a series of Wiener functionals. Since the functionals are estimated implicitly via polynomial kernels, the combinatorial explosion associated with the classical higher-order statistics is avoided. In addition, the kernel framework allows for estimating infinite series expansions and for the regularized estimation of the Wiener series. First results show that image structures such as lines or corners can be predicted correctly, and that pixel interactions up to the order of five play an important role in natural images.

PDF [BibTex]

Franz, M., Schölkopf, B. Implicit Volterra and Wiener Series for Higher-Order Image Analysis In Advances in Data Analysis: Proceedings of the 30th Annual Conference of The Gesellschaft für Klassifikation, 30, pages: 1, March 2006 (inproceedings)

PDF [BibTex]

Model-based Design Analysis and Yield Optimization

Pfingsten, T., Herrmann, D., Rasmussen, C.

IEEE Transactions on Semiconductor Manufacturing, 19(4):475-486, February 2006 (article)

Abstract

Fluctuations are inherent to any fabrication process. Integrated circuits and micro-electro-mechanical systems are particularly affected by these variations, and due to high quality requirements the effect on the devices performance has to be understood quantitatively. In recent years it has become possible to model the performance of such complex systems on the basis of design specifications, and model-based Sensitivity Analysis has made its way into industrial engineering. We show how an efficient Bayesian approach, using a Gaussian process prior, can replace the commonly used brute-force Monte Carlo scheme, making it possible to apply the analysis to computationally costly models. We introduce a number of global, statistically justified sensitivity measures for design analysis and optimization. Two models of integrated systems serve us as case studies to introduce the analysis and to assess its convergence properties. We show that the Bayesian Monte Carlo scheme can save costly simulation runs and can ensure a reliable accuracy of the analysis.

PDF Web DOI [BibTex]

Pfingsten, T., Herrmann, D., Rasmussen, C. Model-based Design Analysis and Yield Optimization IEEE Transactions on Semiconductor Manufacturing, 19(4):475-486, February 2006 (article)

PDF Web DOI [BibTex]

Prenatal development of ocular dominance and orientation maps in a self-organizing model of V1

Jegelka, S., Bednar, J., Miikkulainen, R.

Neurocomputing, 69(10-12):1291-1296, February 2006 (article)

Abstract

How orientation and ocular-dominance (OD) maps develop before visual experience begins is controversial. Possible influences include molecular signals and spontaneous activity, but their contributions remain unclear. This paper presents LISSOM simulations suggesting that previsual spontaneous activity alone is sufficient for realistic OR and OD maps to develop. Individual maps develop robustly with various previsual patterns, and are aided by background noise. However, joint OR/OD maps depend crucially on how correlated the patterns are between eyes, even over brief initial periods. Therefore, future biological experiments should account for multiple activity sources, and should measure map interactions rather than maps of single features.

PDF DOI [BibTex]

Jegelka, S., Bednar, J., Miikkulainen, R. Prenatal development of ocular dominance and orientation maps in a self-organizing model of V1 Neurocomputing, 69(10-12):1291-1296, February 2006 (article)

PDF DOI [BibTex]

Weighting of experimental evidence in macromolecular structure determination

Habeck, M., Rieping, W., Nilges, M.

Proceedings of the National Academy of Sciences of the United States of America, 103(6):1756-1761, February 2006 (article)

Abstract

The determination of macromolecular structures requires weighting of experimental evidence relative to prior physical information. Although it can critically affect the quality of the calculated structures, experimental data are routinely weighted on an empirical basis. At present, cross-validation is the most rigorous method to determine the best weight. We describe a general method to adaptively weight experimental data in the course of structure calculation. It is further shown that the necessity to define weights for the data can be completely alleviated. We demonstrate the method on a structure calculation from NMR data and find that the resulting structures are optimal in terms of accuracy and structural quality. Our method is devoid of the bias imposed by an empirical choice of the weight and has some advantages over estimating the weight by cross-validation.

Web DOI [BibTex]

Habeck, M., Rieping, W., Nilges, M. Weighting of experimental evidence in macromolecular structure determination Proceedings of the National Academy of Sciences of the United States of America, 103(6):1756-1761, February 2006 (article)

Web DOI [BibTex]

Cross-Validation Optimization for Structured Hessian Kernel Methods

Seeger, M., Chapelle, O.

Max-Planck Institute for Biological Cybernetics, Tübingen, Germany, February 2006 (techreport)

Abstract

We address the problem of learning hyperparameters in kernel methods for which the Hessian of the objective is structured. We propose an approximation to the cross-validation log likelihood whose gradient can be computed analytically, solving the hyperparameter learning problem efficiently through nonlinear optimization. Crucially, our learning method is based entirely on matrix-vector multiplication primitives with the kernel matrices and their derivatives, allowing straightforward specialization to new kernels or to large datasets. When applied to the problem of multi-way classification, our method scales linearly in the number of classes and gives rise to state-of-the-art results on a remote imaging task.

PDF Web [BibTex]

Seeger, M., Chapelle, O. Cross-Validation Optimization for Structured Hessian Kernel Methods Max-Planck Institute for Biological Cybernetics, Tübingen, Germany, February 2006 (techreport)

PDF Web [BibTex]

Large-scale prediction of disulphide bridges using kernel methods, two-dimensional recursive neural networks, and weighted graph matching

Cheng, J., Saigo, H., Baldi, P.

Proteins, 62(3):617-629, February 2006 (article)

Web DOI [BibTex]

Cheng, J., Saigo, H., Baldi, P. Large-scale prediction of disulphide bridges using kernel methods, two-dimensional recursive neural networks, and weighted graph matching Proteins, 62(3):617-629, February 2006 (article)

Web DOI [BibTex]

Subspace identification through blind source separation

Grosse-Wentrup, M., Buss, M.

IEEE Signal Processing Letters, 13(2):100-103, February 2006 (article)

Abstract

Given a linear and instantaneous mixture model, we prove that for blind source separation (BSS) algorithms based on mutual information, only sources with non-Gaussian distribution are consistently reconstructed independent of initial conditions. This allows the identification of non-Gaussian sources and consequently the identification of signal and noise subspaces through BSS. The results are illustrated with a simple example, and the implications for a variety of signal processing applications, such as denoising and model identification, are discussed.

PDF Web DOI [BibTex]

Grosse-Wentrup, M., Buss, M. Subspace identification through blind source separation IEEE Signal Processing Letters, 13(2):100-103, February 2006 (article)

PDF Web DOI [BibTex]

Nonstationary Gaussian Process Regression using a Latent Extension of the Input Space

Pfingsten, T., Kuss, M., Rasmussen, C.

pages: 3, January 2006 (poster)

PDF [BibTex]

Pfingsten, T., Kuss, M., Rasmussen, C. Nonstationary Gaussian Process Regression using a Latent Extension of the Input Space pages: 3, January 2006 (poster)

PDF [BibTex]

Classification of Faces in Man and Machine

Graf, A., Wichmann, F., Bülthoff, H., Schölkopf, B.

Neural Computation, 18(1):143-165, January 2006 (article)

PDF Web [BibTex]

Graf, A., Wichmann, F., Bülthoff, H., Schölkopf, B. Classification of Faces in Man and Machine Neural Computation, 18(1):143-165, January 2006 (article)

PDF Web [BibTex]

Gaussian Processes for Machine Learning

Rasmussen, CE., Williams, CKI.

pages: 248, Adaptive Computation and Machine Learning, MIT Press, Cambridge, MA, USA, January 2006 (book)

Abstract

Gaussian processes (GPs) provide a principled, practical, probabilistic approach to learning in kernel machines. GPs have received increased attention in the machine-learning community over the past decade, and this book provides a long-needed systematic and unified treatment of theoretical and practical aspects of GPs in machine learning. The treatment is comprehensive and self-contained, targeted at researchers and students in machine learning and applied statistics. The book deals with the supervised-learning problem for both regression and classification, and includes detailed algorithms. A wide variety of covariance (kernel) functions are presented and their properties discussed. Model selection is discussed both from a Bayesian and a classical perspective. Many connections to other well-known techniques from machine learning and statistics are discussed, including support-vector machines, neural networks, splines, regularization networks, relevance vector machines and others. Theoretical issues including learning curves and the PAC-Bayesian framework are treated, and several approximation methods for learning with large datasets are discussed. The book contains illustrative examples and exercises, and code and datasets are available on the Web. Appendixes provide mathematical background and a discussion of Gaussian Markov processes.

Web [BibTex]

Rasmussen, CE., Williams, CKI. Gaussian Processes for Machine Learning pages: 248, Adaptive Computation and Machine Learning, MIT Press, Cambridge, MA, USA, January 2006 (book)

Web [BibTex]

Class prediction from time series gene expression profiles using dynamical systems kernels

Borgwardt, KM., Vishwanathan, SVN., Kriegel, H-P.

In pages: 547-558, (Editors: Altman, R.B. A.K. Dunker, L. Hunter, T. Murray, T.E. Klein), World Scientific, Singapore, Pacific Symposium on Biocomputing (PSB), January 2006 (inproceedings)

Abstract

We present a kernel-based approach to the classification of time series of gene expression profiles. Our method takes into account the dynamic evolution over time as well as the temporal characteristics of the data. More specifically, we model the evolution of the gene expression profiles as a Linear Time Invariant (LTI) dynamical system and estimate its model parameters. A kernel on dynamical systems is then used to classify these time series. We successfully test our approach on a published dataset to predict response to drug therapy in Multiple Sclerosis patients. For pharmacogenomics, our method offers a huge potential for advanced computational tools in disease diagnosis, and disease and drug therapy outcome prognosis.

PDF Web [BibTex]

Borgwardt, KM., Vishwanathan, SVN., Kriegel, H-P. Class prediction from time series gene expression profiles using dynamical systems kernels In pages: 547-558, (Editors: Altman, R.B. A.K. Dunker, L. Hunter, T. Murray, T.E. Klein), World Scientific, Singapore, Pacific Symposium on Biocomputing (PSB), January 2006 (inproceedings)

PDF Web [BibTex]

Causal Inference by Choosing Graphs with Most Plausible Markov Kernels

Sun, X., Janzing, D., Schölkopf, B.

In Proceedings of the 9th International Symposium on Artificial Intelligence and Mathematics, pages: 1-11, ISAIM, January 2006 (inproceedings)

Abstract

We propose a new inference rule for estimating causal structure that underlies the observed statistical dependencies among n random variables. Our method is based on comparing the conditional distributions of variables given their direct causes (the so-called Markov kernels") for all hypothetical causal directions and choosing the most plausible one. We consider those Markov kernels most plausible, which maximize the (conditional) entropies constrained by their observed first moment (expectation) and second moments (variance and covariance with its direct causes) based on their given domain. In this paper, we discuss our inference rule for causal relationships between two variables in detail, apply it to a real-world temperature data set with known causality and show that our method provides a correct result for the example.

PDF Web [BibTex]

Sun, X., Janzing, D., Schölkopf, B. Causal Inference by Choosing Graphs with Most Plausible Markov Kernels In Proceedings of the 9th International Symposium on Artificial Intelligence and Mathematics, pages: 1-11, ISAIM, January 2006 (inproceedings)

PDF Web [BibTex]

Dimension Reduction as a Deflation Method in ICA

Zhang, K., Chan, L.

IEEE Signal Processing Letters, 13(1):45-48, 2006 (article)

Web [BibTex]

Zhang, K., Chan, L. Dimension Reduction as a Deflation Method in ICA IEEE Signal Processing Letters, 13(1):45-48, 2006 (article)

Web [BibTex]

Extensions of ICA for Causality Discovery in the Hong Kong Stock Market

Zhang, K., Chan, L.

In Neural Information Processing, 13th International Conference, ICONIP 2006, pages: 400-409, (Editors: I King and J Wang and L Chan and D L Wang), Springer, 13th International Conference on Neural Information Processing (ICONIP), 2006, Lecture Notes in Computer Science, 2006, Volume 4234/2006 (inproceedings)

Web DOI [BibTex]

Zhang, K., Chan, L. Extensions of ICA for Causality Discovery in the Hong Kong Stock Market In Neural Information Processing, 13th International Conference, ICONIP 2006, pages: 400-409, (Editors: I King and J Wang and L Chan and D L Wang), Springer, 13th International Conference on Neural Information Processing (ICONIP), 2006, Lecture Notes in Computer Science, 2006, Volume 4234/2006 (inproceedings)

Web DOI [BibTex]

Enhancement of source independence for blind source separation

Zhang, K., Chan, L.

In Independent Component Analysis and Blind Signal Separation, LNCS 3889, pages: 731-738, (Editors: J. Rosca and D. Erdogmus and JC Príncipe und S. Haykin), Springer, Berlin, Germany, 6th International Conference on Independent Component Analysis and Blind Signal Separation (ICA), 2006, Lecture Notes in Computer Science, 2006, Volume 3889/2006 (inproceedings)

Web DOI [BibTex]

Zhang, K., Chan, L. Enhancement of source independence for blind source separation In Independent Component Analysis and Blind Signal Separation, LNCS 3889, pages: 731-738, (Editors: J. Rosca and D. Erdogmus and JC Príncipe und S. Haykin), Springer, Berlin, Germany, 6th International Conference on Independent Component Analysis and Blind Signal Separation (ICA), 2006, Lecture Notes in Computer Science, 2006, Volume 3889/2006 (inproceedings)

Web DOI [BibTex]