A least-squares framework for Component Analysis

People

Fernando de la Torre

Abstract

Over the last century, Component Analysis (CA) methods such as Principal Component Analysis (PCA), Linear Discriminant Analysis (LDA), Canonical Correlation Analysis (CCA), Locality Preserving Projections (LPP), and Spectral Clustering (SC) have been extensively used as a feature extraction step for modeling, classifiation, visualization, and clustering. CA techniques are appealing because many can be formulated as eigen-problems, offering great potential for learning linear and non-linear representations of data without local minima. However, the eigen-formulation often conceals important analytic and computational drawbacks of CA techniques, such as solving generalized eigen-problems with rank deficient matrices (e.g. small sample size problem), lacking intuitive interpretation of normalization factors, and understanding commonalities and differences between CA methods. This paper proposes a unified least-squares framework to formulate many CA methods. We show how PCA, LDA, CCA, LPP, SC, kernel and regularized extensions, correspond to a particular instance of least-squares weighted kernel reduced rank regression (LS-WKRRR). The LS-WKRRR formulation of CA methods has several benefits: (1) provides a clean connection between many CA techniques and an intuitive framework to understand normalization factors; (2) yields efficient numerical schemes to solve CA techniques and overcomes the small sample size problem; (3) provides a framework to easily extend CA methods. We derive weighted generalizations of PCA, LDA, SC and CCA, and a several new CA techniques. In addition, a major appeal of the work is the use of compact matrix formulation

Citation

Fernando de la Torre
"A Least-Squares Framework for Component Analysis",
IEEE Transactions Pattern Analysis and Machine Intelligence (PAMI), 2012
[PDF] [BibTex]

Fernando de la Torre
"A Unification of Component Analysis Methods",
Handbook of Pattern Recognition and Computer Vision (4th edition), World Scientific Publishing Co.,
October 2009

Acknowledgements and Funding

This material is based upon work supported by the U.S. Naval Research Laboratory under Contract No. N00173-07-C-2040. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the U.S. Naval Research Laboratory. Thanks to Louis-Philippe Morency, Chris Ding, Andrew Fitzgibbon, Feng Zhou, Tomas Simon, Minyoung Kim, Karim Abou-Moustafa, Zaid Harchaoui, Jordi Soler for helpful comments and discussions. Thanks to the anonymous reviewers for pointing out to related work and useful comments.

Copyright notice

The documents contained in these directories are included by the contributing authors as a means to ensure timely dissemination of scholarly and technical work on a non-commercial basis. Copyright and all rights therein are maintained by the authors or by other copyright holders, notwithstanding that they have offered their works here electronically. It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.

Human Sensning Lab