Adapting Constrained Markov Decision Process for OCPC Bidding with Delayed Conversions

Nowadays, optimized cost-per-click (OCPC) has been widely adopted in online advertising. In OCPC, the advertiser sets an expected cost-per-conversion and pays per click, while the platform automatically adjusts the bid on each click to meet advertiser’s constraint. Existing bidding methods are based on feedback control, adjusting bids to keep the current cost-per-conversion close to the expected cost-per-conversion to avoid compensation. However, they overlook the conversion lag phenomenon: There always exists a time interval between the ad’s click time and conversion time. This interval makes existing methods overestimate the cost-per-conversion and results in over conservative bidding policies which finally hurts the revenue. To address the issue, this article proposes a novel bidding method, Bidding with Delayed Conversions (Bid-DC) which predicts the conversion probability of the clicked ads and used it to adjust the cost-per-conversion values. To ensure the bidding model can satisfy the advertiser’s constraint, constrained Markov decision process (CMDP) is adapted to automatically learn the optimal parameters from the log data. Both online and offline experiments demonstrate that Bid-DC outperforms the state-of-the-art baselines in terms of improving revenue. Empirical analysis also showed Bid-DC can accurately estimate the cost-per-conversion and make more stable bids.