Testing False Recalls in E-Commerce Apps: a User-Perspective Blackbox Approach

Search components are essential in e-commerce apps. However, they often suffer from bugs, leading to false recalls, i.e., irrelevant search results. We propose false recall Hound (frHound), a black box testing approach targeting false recalls, despite the challenge induced by ambiguous natural language description. frHound mimics how users make purchasing decisions during online shopping and detects the most divergent search results, which are likely false recalls, as most search results are relevant during e-commerce searches. It designs <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{3 7}$</tex> features to align with how users process information during online shopping and uses an outlier detection technique to identify the most divergent search results. Experiments with real industry data show frHound reduces human labor, time, and financial costs associated with discovering false recalls by 36.74 times. In a seven-month trial with M-app, a popular Chinese e-commerce platform, frHound identified 1282 false recalls, improving user satisfaction and reducing false recall discovery costs.