Nonparametric Bayesian Models for Supervised Dimension Reduction and Regression

Abstract

We propose nonparametric Bayesian models for supervised dimensionreduction and regression problems. Supervised dimension reduction isa setting where one needs to reduce the dimensionality of thepredictors or find the dimension reduction subspace and lose littleor no predictive information. Our first method retrieves thedimension reduction subspace in the inverse regression framework byutilizing a dependent Dirichlet process that allows for naturalclustering for the data in terms of both the response and predictorvariables. Our second method is based on ideas from the gradientlearning framework and retrieves the dimension reduction subspacethrough coherent nonparametric Bayesian kernel models. We alsodiscuss and provide a new rationalization of kernel regression basedon nonparametric Bayesian models allowing for direct and formalinference on the uncertain regression functions. Our proposed modelsapply for high dimensional cases where the number of variables farexceed the sample size, and hold for both the classical setting ofEuclidean subspaces and the Riemannian setting where the marginaldistribution is concentrated on a manifold. Our Bayesian perspectiveadds appropriate probabilistic and statistical frameworks that allowfor rich inference such as uncertainty estimation which is importantfor measuring the estimates. Formal probabilistic models withlikelihoods and priors are given and efficient posterior samplingcan be obtained by Markov chain Monte Carlo methodologies,particularly Gibbs sampling schemes. For the supervised dimensionreduction as the posterior draws are linear subspaces which arepoints on a Grassmann manifold, we do the posterior inference withrespect to geodesics on the Grassmannian. The utility of ourapproaches is illustrated on simulated and real examples.