Orchestrated columnar-based analysis with <i>columnflow</i>

Columnflow is a tool for columnar-based data analysis. It is written in Python , experiment-agnostic in its core, and supports any flat n -tuple format, such as ROOT -based TTrees or Parquet files. Leveraging the vast Python ecosystem, vectorization and convenient representation of event content can be achieved through NumPy , AwkwardArray and other libraries. Based on the Luigi Analysis Workflow (law) package, columnflow provides full analysis automation over arbitrary, distributed computing resources. This approach features persistent, intermediate outputs for purposes of reusing of previously computed results, debugging, and exchange with collaborators. Job submission to various batch systems is natively supported. Remote files can be seamlessly accessed via various protocols using the Grid File Access Library (GFAL2). In addition, a sandboxing mechanism can encapsulate the execution of parts of a workflow in dedicated environments, supporting subshells, Python virtual environments, and containers. This contribution introduces the key components of columnflow and highlights the benefits of a fully automated workflow for complex and large-scale HEP analyses.