Supporting documents (software documentation) for manuscript: Leveraging Social Media Data to Study Disease and Treatment Characteristics of Hodgkin's Lymphoma Using Natural Language Processing Methods

This folder contains code associated with the manuscript: "Leveraging Social Media Data to Study Disease and Treatment Characteristics of Hodgkin’s Lymphoma Using Natural Language Processing Methods. project involves:accessing Twitter (X) data via the X API.cleaning and preprocessing the acquired tweets and using NLP methods descibed in the manuscriptannotating text data manually and performing adjudication between annotators using e-Host.visualizing results using R and ggplot. ## Step 1: Accessing X (Twitter) Data Files & Folders Involved: `X_data_api.R``lib_all` (R library file containing required packages and functions) instructions:1. obtain a Bearer Token from X (Twitter Developer Portal). You must have API access granted by X. 2. `lib_all` file can be sourced to load all necessary libraries.3. insert your unique Bearer Token into the specified variable in `X_data_api.R`.4. run the code to connect to the X API, specificy the date ranges, modify the query as needed. ### Step 2: Data Cleaning and Preprocessing Files & Folders Involved: `text_processing` scripts Python script Instructions:1. Refer to the `text_processing` scripts for detailed steps on cleaning the raw tweet data: lowercasing and removing unwanted characters (URLs, removing non-English characters, etc.) Convert tweets to individual texts and savie them as individual text files.2. Also, create `df_nlp` (created after preprocessing) to: apply NLTK-based NLP routines (e.g., tokenization, POS tagging, lemmatization, named entity recognition) approximate string matching to identify specific keywords from your predefined lists ### Step 3: Annotation and Adjudication note: for creating the pre-defined list used the e-Host tool by identifying and annotating the text. Files & Folders Involved: `XML` folder (contains XML files from annotation tools) `adjudication_files` folder (contains adjudicated annotation results) `adjudication_files\annotator2\lib\test\adjudication` (corpus files from adjudication) Instructions:1. Use the XML annotation files to understand how tweets were manually annotated for disease and treatment characteristics.2. If needed, open the adjudication files to see how discrepancies between two annotators’ labels were resolved.3. The `adjudication` folder contains the sample of corpus after annotations ### Step 4: Visualization Using R (ggplot) **Files & Folders Involved:** - `graphs` folder r codes for ggplot Use provided R code, adjust according to your outcomes to get the graph plots similar to what presented in the manuscript. ### note: XML and adjudication_files are uploaded as zip folder.