More Than Meets the Eye: On Evaluating SBOM Tools In Java

Open-source software is widely used in current software development. Unfortunately, this integration introduces a spectrum of challenges and potential threats. Such challenges emerge due to the diversity of import scenarios, which in turn may introduce malicious or vulnerable code in the client software, thereby causing significant security risks. To improve the transparency of software supply chains, Software Bill of Materials (SBOM) tools are proposed to identify the components within software systems. However, there limit investigation of functionality (i.e., tool operational process and data fields) and their practical performance of SBOM tools across various import scenarios. In this paper, we perform a comprehensive empirical study to investigate the impact of different import scenarios on SBOM tools. Specifically, we focus on three distinct component import scenarios: Build Tool Import, Dynamic Loading, and Source Code Import across a new benchmark consisting of 152 projects. We find that (1) The detection capabilities of SBOM tools exhibit considerable variance, especially in identifying dependency relationships; (2) The effectiveness of SBOM tools within the import scenarios of Dynamic Loading and Source Code Import falls short of expectations. Based on our findings, we summarize the lessons learned from different perspectives, including practitioners, tool vendors, and researchers. Our study provides valuable insights into the intricate landscape of software component usage, contributing to enhancing SBOM tools in modern software development.