Unsupervised Risk Estimation Using Only Conditional Independence Structure

We show how to estimate a model’s test error from unlabeled data, on distributions very different from the training distribution, while assuming only that certain conditional independencies are preserved between train and test. We do not need to assume that the optimal predictor is the same between train and test, or that the true distribution lies in any parametric family. We can also efficiently compute gradients of the estimated error and hence perform unsupervised discriminative learning. Our technical tool is the method of moments, which allows us to exploit conditional independencies in the absence of a fully-specified model. Our framework encompasses a large family of losses including the log and exponential loss, and extends to structured output settings such as conditional random fields.

Authors

• Jacob Steinhardt
• Percy Liang Department of Computer Science, Stanford School of Engineering, Stanford, CA, USA Human-Centered Artificial Intelligence Institute, Stanford University, Stanford, CA, USA

Keywords

No keywords are indexed for this paper.