Statistical Designs for Learning User Preference and Experience

For any organization, designing a new product or planning the features of a new service is a complex decision-making process which depends on understanding its users' preferences and the experiences they have with prototypes of the new product or service. Design of experiments (DOE) offers a framework to aid in the modeling and collection of data to learn user preferences and experiences to facilitate this decision-making process. Questions such as ``which of our newly proposed products is preferred most by our customers" and ``which version of our new service results in the highest monthly revenue" can be readily addressed by carefully designing and analyzing an appropriate experiment. This dissertation focuses on the design of experiments for learning user preference and experience, and consists of four projects related to experimental design where product or service users are involved: (1) Bayesian sequential preference elicitation; (2) Batch sequential designs in Bayesian preference elicitation; (3) Approximate dynamic programming methods in Bayesian preference elicitation and (4) Collaborative design of controlled experiments in the presence of subject covariates. In (1), we introduce the framework of Bayesian sequential preference elicitation, which is the foundation for the developments in (2) and (3). In (2), we extend (1) to the case where multiple queries are selected at once. In (3), we investigate non-greedy approximate dynamic programming methods in the context of preference elicitation. Lastly, in (4) we propose methodology and algorithms for the experimental design of a certain A/B testing problem.