Generative Modeling of Dependencies for Features, Instances and Distributions

Modeling the dependencies presented in data is fundamental for many machine learning tasks. In this dissertation, we develop models to capture the dependencies between subsets of features, between instances within a neighborhood, and between sets of partially observed instances. We propose an unified framework for different types of the dependencies by modeling the conditional distribution of arbitrary subsets. For example, the dependencies across features can be modeled by the distribution of an arbitrary subset of features conditioned on another subset of features. Similarly, the dependencies across instances can be captured by the distribution for a set of instances conditioned on another set of relevant instances, such as the prototypes of the corresponding neighborhood. By modeling the conditional distribution for a set of partially observed instances, we can capture the dependencies cross both features and instances jointly. The estimated conditional distributions enable us to further analyze and exploit the underlying dependencies for downstream tasks. Here, we leverage the conditionals to assess the informativeness of candidate features before even acquiring their values. Since acquiring the feature values could incur tremendous costs in many real-world scenarios, our framework offers a possibility to reduce the feature acquisition costs by acquiring only the necessary information for the task. We extend past static ML and develop an active feature acquisition (AFA) framework that interacts with the environment to obtain new information on-the-fly and can: 1) make inferences on an instance in the face of incomplete features, 2) determine a plan for feature acquisitions to obtain additional information on the instance at hand. To take a step further, we extend the AFA framework to sequential decision making scenarios, where each decision is made taking into account the cost of feature acquisitions. We build our model on a backbone of understanding the dependencies of features across time steps. This dissertation makes contributions in the following two aspects. First, we build generative models that capture dependencies over features, instances and distributions. Second, we exploit the learned dependencies to address cost-sensitive problems.