Adversarial distractor generation for MCQA: Leveraging in-context learning and rule-based approaches

• This study proposes in-context learning and rule-based techniques to enhance MCQA systems by creating challenging distractors. • This study utilizes a prompt-based strategy to generate diverse and complex distractors. • This study introduces a rule-based method adding misleading values to answer options to test model resilience. • Experiments are conducted on the T5-Base and RoBERTa-Large, showing consistent performance declines across three MCQA datasets. • This study highlights vulnerabilities in MCQA systems, offering insights for future NLP advancements. Adversarial distractor generation strengthens the capability of Multiple Choice Question Answering (MCQA) systems. This study introduces two distinct methods: in-context learning and rule-based techniques, which are designed explicitly for adversarial distractor generation for MCQA datasets. We leverage the LLama-7b model in the in-context learning approach, following a prompt-based strategy for adversarial distractor generation. This methodology is designed to enrich the variety and complexity of the distractors. Besides, this study presents a rule-based approach, mainly focusing on the impact of appending garbage values to the answer options. Via comprehensive experiments, we systematically evaluate the results of two approaches on the T5-Base and RoBERTa-Large across three MCQA datasets. The empirical results demonstrate a constant decrease in performance across all evaluated datasets when utilizing both the in-context learning and rule-based approaches.