Comprehensibility and Acceptance of Language Workbenches: a Family of Experiments with Neverlang - Replication Package

NEVERLANG STUDY REPLICATION PACKAGEThe present folder contains the replication package for the paper "Comprehensibility and Acceptance of Language Workbenches: a Family of Experiments with Neverlang" by Giovanna Broccia, Alessio Ferrari, Maurice ter Beek, Walter Cazzola, Luca Favalli, Francesco Bertolotti.Language workbenches are tools that enable the definition, reuse and composition of programming languages and their ecosystems, aiming to streamline language development. To facilitate their adoption by language designers, the comprehensibility of the meta-language (i.e., the language used to define other languages) used in these workbenches is an important aspect to consider and evaluate. Moreover, considering that language workbenches are relatively new tools and still not widely widespread, user acceptance emerges as a crucial factor to be accounted for in their assessment. Current literature often neglects user-centred aspects like comprehensibility and acceptance in the assessment of this breed of tools. This paper addresses this gap through a family of experiments assessing Neverlang, a modular language workbench. Our experiments consider the comprehensibility of Neverlang’s language and programs, alongside user acceptance in terms of ease of use, usefulness, and intention to use. We also explore the relationship between Neverlang comprehensibility and users’ working memory capacity. The results reveal that users possess a satisfactory comprehension of Neverlang, particularly regarding syntax, and express favourable perceptions of its usefulness and intention to use. Surprisingly, no significant correlation is found between Neverlang’s comprehensibility and user acceptance, nor between users’ working memory capacity and their comprehension. Additionally, while Neverlang’s ease of use is not positively perceived, the intention to use the tool shows a clear correlation with both its perceived usefulness and ease of use. This study underscores the importance of considering both technical and user-centred aspects in evaluating language workbenches. Our findings show the limited role played by working memory capacity in the comprehension of the meta-language of the considered language workbench and the primary role of perceived usefulness in facilitating its practical adoption. The present folder contains the instruction and the material to perform the study and all the data to check the results presented in the paper. Additionally, the folder contains the txt file with the license (CC BY-NC 4.0). In particular, the folder contains the following sub-folders: 1. Training Materials The folder contains The slides used to create the training video on Neverlang used in the original experiment and in the second replication (ITA). The video is available at https://youtu.be/q3E9MxmtOXE?si=mB6K_oei64_DDqQF. The slides used for the training phase in the second replication. 2. Neverlang Comprehensibility test The folder contains the PDF (not editable) version of the test, composed of the questions on all three cognitive dimensions (learnability, understandability, and evolving). The original (editable) version of the questionnaire is available at the following links (evolving questions are not available in English since have been used only in the second replication of the experiment in Italian. For the English version, please check the PDF version of the test): - Learnability and Understandability questions (ITA) https://it.surveymonkey.com/r/BBRL63P - Learnability and Understandability questions (ENG) https://it.surveymonkey.com/r/R5N8GMJ - Evolving questions (ITA) https://it.surveymonkey.com/r/2NT32DJ 3. Post-test questionnaire The folder contains the PDF (not editable) version of the post-test questionnaire. The original (editable) version of the questionnaire is available at the following links: - https://docs.google.com/forms/d/e/1FAIpQLSdF7jnBp_FdyVqgLil9__ARdENboqvESZVNIydgb-c0yg9E3A/viewform?usp=sf_link (ENG) - https://docs.google.com/forms/d/e/1FAIpQLSfXz-8ioi1HUbcStqcTz3RpMxmmqugx8a85AjgxgTZZxHXsGA/viewform?usp=sf_link (ITA) 4. Working Memory Span Tasks web application The folder contains two subfolders containing the code to run the web application for the working memory span tasks (one for the Italian version used in the original experiment and in the second replication and one for the English version used in the second replication). The web application is also available at the following links: - https://wmspantasks.isti.cnr.it/ (ITA) - https://wmspantasks.isti.cnr.it/ENG.html (ENG) 5. Data analysis The folder contains the following files: Total_WMcapacityScores (excel file with the scores for the Working Memory Span Task for all experiments) BackgroundInfo_OriginalExperiment (excel file with the background information collected during post-test questionnaire in the Original Experiment - UniMi 1 2022) BackgroundInfo_FirstReplication(excel file with the background information collected during post-test questionnaire in the First Replication - BISS 2023) BackgroundInfo_SecondReplication (excel file with the background information collected during post-test questionnaire in the Second Replication - UniMi 2 2023) and the following sub-folders: Data_analysis_Original_UniMi1 Data_analysis_FirstReplication_BISS Data_analysis_SecondReplication_UniMi2 Meta-analysis of proportion Each of the sub-folders contains: The data to perform the analysis (.csv) The R file to run the analysis (.rmd) The PDF document with the results of the analysis INSTRUCTION TO RUN THE ANALYSIS WITH R: 1. Download and install R from here: https://cran.rstudio.com/ 2. Download and install RStudio Desktop (Open Source Edition) from here: https://posit.co/download/rstudio-desktop/ --- ATTENTION: Always install the required additional packages if prompted --- 3. Open the .rmd file (sub-folder 6. Data analysis) with RStudio ("File > Open File"). The tool will display the analysis file in a window. Make sure that the .rmd file and the .csv file(s) on which the analysis is performed are in the same folder.4. Run the analysis (can be performed in two ways): I. Once for all code chunks: click "Run > Run All" (top right row of the window). II: Chunk by chunk: click "Run Current Chunk" (right of the code chunk in the window). Make sure to click "Run All Chunks Above" if previous chunks have not been run previously (left of the "Run Current Chunk" in the window). 5. Visualise the results (can be visualised in two ways): I. In a textual form: the tool will show the textual results in a separate window. II. In a separate HTML, PDF, or Word file with plots: click the "Knit" button in the analysis window. The file will be saved in the .rmd file directory.