Determining Code Proficiency Levels from Python Textbooks

The ability to measure developer proficiency is crucial, as it reflects an individual's capability to understand and interpret efficient, effective, and well-structured code. It is an essential aspect of ensuring software quality. Although prior work has proposed different approaches for measuring proficiency through code, the process of learning various coding concepts remains non-trivial and highly debated. This paper proposes a framework for determining code proficiency through learning aids that serve as ground-truth—namely, textbooks—and two automated methods: Übersequence and Clustering. We conducted an empirical study to identify Python code proficiency levels. We then assessed the effectiveness of the framework by addressing two key research questions. Using a dataset of 22 introductory Python textbooks and code constructs referenced from Python AST , we achieved a high coverage of 85.51% for Python code constructs. Our analysis reveals a significantly high similarity in the sequence of code construct introductions across the textbooks, supporting our methodology of using textbooks to assess code proficiency. The resulting Übersequence demonstrates the feasibility of assigning proficiency levels to individual code constructs, while clustering further enables a structured grouping perspective. To demonstrate practical applicability, we present examples and initiate discussions on future research directions, particularly software maintenance tasks such as bug assignment and code reviews.