Towards Realistic Results for Instrumentation-Based Profilers for JIT-Compiled Systems

Profilers are crucial tools for identifying and improving ap- plication performance. However, for language implementa- tions with just-in-time (JIT) compilation, e.g., for Java and JavaScript, instrumentation-based profilers can have signifi- cant overheads and report unrealistic results caused by the instrumentation. In this paper, we examine state-of-the-art instrumentation- based profilers for Java to determine the realism of their results. We assess their overhead, the effect on compilation time, and the generated bytecode. We found that the pro- filer with the lowest overhead increased run time by 82×. Additionally, we investigate the realism of results by test- ing a profiler’s ability to detect whether inlining is enabled, which is an important compiler optimization. Our results document that instrumentation can alter program behavior so that performance observations are unrealistic, i.e., they do not reflect the performance of the uninstrumented program. As a solution, we sketch late-compiler-phase-based in- strumentation for just-in-time compilers, which gives us the precision of instrumentation-based profiling with an over- head that is multiple magnitudes lower than that of standard instrumentation-based profilers, with a median overhead of 23.3% (min. 1.4%, max. 464%). By inserting probes late in the compilation process, we avoid interfering with compiler optimizations, which yields more realistic results.