Algorithmic Decision‐Making Safeguarded by Human Knowledge

ABSTRACT Commercial AI solutions provide analysts and managers with data‐driven business intelligence for a wide range of decisions, such as demand forecasting and pricing. However, human analysts may have their own insights and experiences about the decision‐making that are at odds with the algorithmic recommendation. In light of such a conflict, we study problems in which humans and AI interact in the decision‐making process and characterize the conditions under which human knowledge adds value to AI decision‐making. In this paper, we provide a general analytical framework for studying the augmentation of algorithmic decisions with human knowledge: the analyst uses the knowledge to set a guardrail by which the algorithmic decision is clipped if the algorithmic output is out of bounds and appears unreasonable. We show that when the algorithmic decision is asymptotically optimal with large data, the non‐data‐driven human guardrail usually provides no benefit. However, we point out two common pitfalls of the algorithmic decision: (1) lack of domain knowledge, such as market competition, and (2) model misspecification. In these cases, even with sufficient data, the augmentation from human knowledge can still improve the performance of the algorithmic decision. We propose a model to capture a practical and pervasive type of human–AI interaction in the decision‐making process. We derive insights into when the human analyst should follow the algorithmic recommendation. We conclude that even in the era of big data, human knowledge can still play an essential role in decision‐making.