An Analytical Study of a Governed Lakehouse Framework in Multi-State Medicare Claims and Provider Data for Fraud, Waste, and Abuse Detection

Medicare fraud detection, waste, and abuse is hampered by the high volume of claims, highly fragmented data-analysis systems, limited visibility into data quality, and significant class imbalance. This study assessed the potential of an Original Medicare Part B (OMB) Bronze-Silver-Gold lakehouse to improve data quality, traceability, analytical efficiency, and provider-level risk identification across OMB-covered populations in California, New York, Florida, Texas, and Pennsylvania in 2024. Using Apache Spark/PySpark with Delta-format tables, the study sampled 1,500,000 provider-service records in proportion to state Traditional Medicare enrollment, representing 214,873 unique NPIs, consolidated them with CMS provider files and HHS-OIG provider-exclusion data, interpreted 30 features, and used four models: Logistic Regression, Random Forest, XGBoost, and Isolation Forest. During Silver-layer screening, 27,362 records (1.82%) triggered one or more quality rules and were quarantined; NPI enrichment had 99.7% compliance, and Gold-layer data quality had 99.2% compliance. This reduced query time from a median of 18.4 seconds to 5.7 seconds (approximately 69%). XGBoost performed best among the supervised models, with ROC-AUC = 0.91, PR-AUC = 0.34, and F1 = 0.66. State elevated/high-risk flags were not confirmed as fraud; they were a priority for investigation and ranged from 2.2%–3.4% of unique providers within each state. Manually managed, repeatable pipelines provide defensible Medicare FWA prioritization, and repeatable, source-to-feature, auditable pipelines can support multi-state, analytic workflows where needed. Areas to explore include longitudinal claims, Part D, DMEPOS, enforcement outcomes, graph analytics, and prospective investigator validation.