PROBE4RSE: Provenance Replay/Observation Engine for Research Software Engineers

Have you ever asked, “did this output use the right version of the inputs and code?”, “what software does this program require to execute again?”, “how can I convert this pile-of-scripts to a containerized workflow?”, “how does the ancestry of these two outputs differ?“, “who is using my software library?”, or a similar question. All these are example questions that computational provenance can help answer. Computational provenance is the process by which a certain computational artifact was generated, including its inputs (e.g., data, libraries, executables) and the computational provenance of those inputs (Friere et al. 2008). How to collect computational provenance? We could ask the application developers to emit this kind of data. That approach requires a herculean effort to get all applications to comply. We could require the user to use workflow systems that explicitly declare the inputs and outputs of every node. This approach shifts the compliance burden onto the user. If the user mis-specifies the workflow, it may still execute but the provenance would be wrong. The “holy-grail” would be to collect provenance data at the system-level without modifying any application code and not needing superuser privileges or harming performance. Almost all prior attempts at unprivileged system-level provenance collection used ptrace syscall, which asks the Linux kernel to switch to the tracing process every time the tracee process executes a system call (like how strace binary works). Ptrace-based tracers meet most of the technical requirements but are prohibitively slow. Our recently accepted work (Grayson et al. 2024) observed a geometric mean of traced runtime 1.5x for CARE (Janin et al. 2014), 2.5x for RR (O’Callahan et al. 2017), and 3x for ReproZip (Chirigati et al. 2016) over the untraced runtime. Ptrace involves a context-switch from the tracee process to the tracer process and back every system call, of which there could be thousands per second. Each context switch causes scheduler overhead and clears caches (especially the translation lookaside buffer). We propose PROBE (Provenance for Replay OBservation Engine), a tool that collects system-level provenance using library interpositioning (Curry 1994), also known as the LD_PRELOAD trick. Library interpositioning happens all within the same virtual address space, which does not involve any additional context switching. We have a working research prototyp operating this way. Another possible reason system-level provenance hasn’t caught on is the lack of downstream tooling. We are developing several consumers of provenance including a graphical viewer, an automatic containerization tool, an environment “diff”, and a software citation generator. Furthermore, we export our provenance to Process Run RO Crate format (Leo et al. 2023), so that it can be interoperable with other provenance consumers.