Automated vs Manual Prompt Optimisation for Legal Contract Clause Extraction: A Comparative Study of DSPy and Few-Shot Prompting Using LLM-Based Evaluation

This study compares Manual Few-Shot promptingversus DSPy (Declarative Self-improving Python)for legal contract clause extraction. We evaluated both approaches on 50 real SEClegal contracts from the CUAD benchmark datasetacross 4 clause types: Parties, Governing Law,Expiration Date, and Effective Date. Training set: 10 contractsTest set: 20 contractsEvaluation: Claude Haiku as LLM judge (0-3 scale) Results show DSPy outperforms manual few-shotoverall (1.94 vs 1.85 average score, +4.9%). Key finding: DSPy shows largest improvement onGoverning Law (+21%) — the most complex clause.Manual few-shot wins on Effective Date —the simplest formulaic clause. Practical insight: DSPy beats manual promptingon complexity. Human intuition wins on simplicity.