Artifact of Fuzzing the Guts of Machine Learning

README This directory contains files related to fuzzing, coverage analysis, and setup, organised into two sections: Code/Tools/Environment and Results. Code, Tools, and Environment 0. Full copy of our code at GitHub: fuzzingML-main.zip. 1. fuzzingML-main.zip and Experiments_Scripts_for_Python-diff-testing-N-coverage.zip (Python, GreyBox) - First zip contains the source code for the tool, a fuzzer based on machine-learning techniques. - Second zip contains the scripts to build Python3 with coverage and to run differential testing. 2. afl_scriptsNseeds4CrashBaseTest.tar.gz (Python, GreyBox) - Includes scripts and seed files required to start the fuzzing process. 3. m510_setup.tar.gz (Python, GreyBox) - The complete fuzzing environment as it was tarred from the `m510` machine. - Setup all in /opt/ and run AFL++ in hot-storage for `m510` machine. user@node:/dev/shm$ dd if=/dev/zero of=testfile bs=1G count=1 oflag=dsync 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.866424 s, 1.2 GB/s 4. Build_cov_env_cpython_scripts.tat.gz (Python, GreyBox) - Scripts to build on your own machine, 5-6 parts for coverage.5. cpython-wth-mlighter.tar.gz (Python, GreyBox) - Instrumented version of CPython with `gcov` for coverage analysis, along with installed `sklearn` version 1.6.0. Seed test_916698062 was tested with `sklearn` version 1.5.2 due to a change of function interface. - To restore the installation of Mlighter util, you will need to run: cd /users/user42/cpython/install/usr/local/bin ./../lib/mlighter/mlighter/utils/install.sh 6. coverage_env_wt_gfauto.tar.gz (Python, GreyBox) - The coverage environment was prepared and tarred, containing the instrumented CPython setup. - Packages installed for Coverage machine: gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 and in general on the machine: Package Version ---------------------- ------------- blinker 1.4 build 1.2.2.post1 certifi 2020.6.20 command-not-found 0.3 coverage 7.6.10 cryptography 3.4.8 dbus-python 1.2.18 distlib 0.3.9 distro 1.7.0 distro-info 1.1build1 filelock 3.16.1 httplib2 0.20.2 importlib-metadata 4.6.4 jeepney 0.7.1 joblib 1.4.2 keyring 23.5.0 launchpadlib 1.10.16 lazr.restfulclient 0.14.4 lazr.uri 1.0.6 mlighter-utils 1.1.1 more-itertools 8.10.0 netifaces 0.11.0 numpy 2.2.0 oauthlib 3.2.0 packaging 24.1 pandas 2.2.3 pip 24.3.1 pipenv 2024.4.0 platformdirs 4.3.6 Pygments 2.11.2 PyGObject 3.42.1 PyJWT 2.3.0 pyparsing 2.4.7 pyproject_hooks 1.2.0 python-apt 2.4.0+ubuntu1 python-dateutil 2.9.0.post0 pytz 2024.2 PyYAML 5.4.1 scikit-learn 1.6.0 scipy 1.14.1 SecretStorage 3.3.1 setuptools 75.6.0 six 1.17.0 ssh-import-id 5.11 threadpoolctl 3.5.0 tomli 2.0.2 tzdata 2024.2 ubuntu-advantage-tools 8001 ufw 0.36.1 virtualenv 20.28.0 virtualenv-clone 0.3.0 wadllib 1.3.6 wheel 0.37.1 zipp 1.0.0 and in the instrumented Cpython version: Package Version --------------- ----------- build 1.2.2.post1coverage 7.6.10 distlib 0.3.9 filelock 3.16.1 joblib 1.4.2 mlighter-utils 1.1.1 numpy 2.2.0 packaging 24.2 pandas 2.2.3 pip 24.3.1 pipenv 2024.4.0 platformdirs 4.3.6 pyproject_hooks 1.2.0 python-dateutil 2.9.0.post0 pytz 2024.2 scikit-learn 1.6.0 scipy 1.14.1 setuptools 75.6.0 six 1.17.0 threadpoolctl 3.5.0 tomli 2.2.1 tzdata 2024.2virtualenv 20.28.0xgboost 2.1.3 Except for seed 902236794 for which we used numpy 2.2.1scikit-learn 1.5.2due to compatbility issues. For all seeds, we uesdsetuptools 65.5.0 when taking the coverage for the Python part (it seems like a minor change, so we left it as is). To run the base coverage with the original seed generated alongside with the template, run the following command: seeds=(137431083 374268032 375431503 475093897 523744128 523989622 649447714 708003241 754757302 758989938 770797067 809881512 86784155 88885636 902236794 916698062)for myseed in "${seeds[@]}"; do mkdir -p /users/user42/testArchive/outputs_exp_"$myseed"_0/default/queue/; cp Experiments/test_"$myseed"/__test_input_"$myseed".npy /users/user42/testArchive/outputs_exp_"$myseed"_0/default/queue/; doneThen run regularly the coverage script: ./2-collect-all-coverage.sh 7. Differntial testing (Python, GreyBox): - Run the following commands to install the compilers for diff-testing but make sure you have gcc-11 and python3.10 already installed in your machine ./0-setup.she su - user42 # Incase you have problem with ssl try this: sudo apt-get install libssl-dev openssl sudo apt-get install -y libmpdec-dev # The continue with the installation of mlighter: git clone https://github.com/hdg7/mlighter.git cd mlighter sudo ./mlighter/utils/install.sh ./1-python-plain-build.sh then make sure the versions are relatively the same, as some pip modules do not always have support backwards. One such case is sklearn which needs version 1.5.2 for the last seed (seed 16, AdaBoosClassifier). If you still have issues with openssl, you can try this: # openssl wget https://www.openssl.org/source/openssl-1.1.1l.tar.gz tar zxf openssl-1.1.1l.tar.gz cd openssl-1.1.1l ./config --prefix=/users/user42/cpython make make install Then run differential testing: - Run the differential testing between the system Python 3.10 and the locally built Python 3.13. Before running, make sure to have the testArchive and Experiments folders under the home folder of user42 (untar this afl_scriptsNseeds4CrashBaseTest.tar.gz). nohup ./2-diff-testing.sh > report.log 2>&1 &8. Experiments with R (R, GreyBox) - Templates are in the Zip file fuzzingML-main.zip, under the subfolder: R/finalTemplates (TEMPLATES) - Files: outputsAFL1.zip, outputsAFL2.zip and outputsAFL3.zip include the output folder from AFL++ fuzzing with R seeds. (DATA) - To reproduce these results: unzip fuzzingML-main.zip cd fuzzingML-main ## Then Dockerfile is a ready version of our experiments. Build it first: cd R docker build -t dockerfile . ## Then edit the paths /home/ubuntu/dataExtra/fuzzingML/R/finalTemplates:/home/advml/template to be your own and run: ./runScript.bash 9. Experiments with Python Black Box Fuzzing: TODO. - coverageClatest.tar.xz contains the coverage results for the seeds in C. Evaluation Setup We generated 16 Python templates, marked in the paper from 1-16. The template numbers in the data here are as follows: |-----|-----------------|------------------------------------------|| # | Template ID. | Models ||-----|-----------------|------------------------------------------|| 1 | 137431083 | `KNeighborsClassifier` || 2 | 374268032 | `RadiusNeighborsClassifier` || 3 | 475093897 | `LinearRegression` || 4 | 523744128 | `GradientBoostingClassifier` || 5 | 523989622 | `MLPClassifier` || 6 | 649447714 | `GaussianNB and KNeighborsClassifier` || 7 | 708003241 | `Ridge` || 8 | 754757302 | `LogisticRegression` || 9 | 758989938 | `SVC` || 10 | 770797067 | `DecisionTreeClassifier` || 11 | 809881512 | `GaussianNB` || 12 | 86784155 | `RandomForestClassifier` || 13 | 88885636 | `LinearDiscriminantAnalysis` || 14 | 902236794 | `XGBClassifier (from XGBClassifier)` || 15 | 916698062 | `AdaBoostClassifier` ||-----|-----------------|------------------------------------------| You can use the template ID above to locate the specific template in the Experiment folder. Results 1. Results_AFL_crashTesting_24h_31sets_N_12days_1set.tar.gz - Results from the fuzzing process, covering 24-hour runs with 31 sets across 12 days. 2. Results of Coverage: - Coverage of the C/C++ part of the CPython compiler: - Part 1: coverage_c_part_1.tar.gz (9 out of the 11 seeds) - Part 2: coverage_c_part_2.tar.gz (1 out of 11 seeds - repeat) - Part 3: coverage_c_part_3.tar.gz (2 out of 11 seeds) - Coverage summary and analysis of the C part is available here: function_coverage-16seeds_C.xlsx and line_coverage-16seeds_C.xlsx. Usage Overview - Fuzzing Setup: - Use `fuzzingML-main.zip` for the fuzzer implementation. - Extract `afl_scriptsNseeds4CrashBaseTest.tar.gz` to get the necessary scripts and seed files for fuzzing. - Environment: - The fuzzing environment is packaged in `m510_setup.tar.gz`. Extract this to replicate the setup on the `m510` machine. - Coverage Analysis: - Use `cpython-wth-mlighter.tar.gz` for the CPython setup instrumented with `gcov`. - The coverage environment is stored in `coverage_env_wt_gfauto.tar.gz`.