Variance and robustness aware UCB variants for evaluating e-commerce fulfilments

The online allocations of inventory to different product categories are naturally a multi-armed bandit problem. Companies need efficient and low-regret exploration to allocate the deep inventories to optimal options. This research intends to do an offline evaluation of UCB variants on a public e-commerce sales dataset, setting the arms as product categories and binary rewards from the terminal fulfilment status. It is comparing UCB1, Asymptotically Optimal UCB, UCB-Variance, Bootstrapped UCB, and IQR-UCB algorithms for performances in regret, pull allocation, and runtime comparison. The UCB-Variance algorithms result in the optimal performance combination with low regret and baseline runtime, while optimal-UCB offers a slight improvement and the baseline runtime. Bootstrapped UCB offers little gain at high costs, while the IQR- UCB shows overly aggressive exploration on binary rewards due to the vanishing bonus. For the category-level allocation, the UCB-Variance algorithm seems to be the most cost-benefit, and Optimal-UCB is a safe anytime baseline. However, conservative success labelling and less operational results provides rooms for further optimization. We should also evaluate Bootstrapped UCB with upper-quantile and lazy updates, IQR- UCB with alternative scales and adding floors, and non-stationary and contextual settings to better examine the full potentials of these methods.