Next Generation Sequencing (NGS) - Repeatability, Reproducibility and accuracy of commercially available Cancer Panels

Use of a Next Generation Sequencing (NGS)-based test as a clinical trial assay requires that the test be analytically validated to CLIA/CAP regulations. During the course of analytically validating NGS cancer panels for use as patient enrollment assays, we assessed the repeatability, reproducibility and accuracy of commercially available Cancer Panels (Illumina TruSeq Cancer Panel on MiSeq and Life Technologies AmpliSeq v2 Cancer Panel on Ion Torrent PGM). We measured the repeatability and reproducibility by evaluating all variant calls among technical replicates and found in both platforms that variants with higher variant frequency (VF >30%) were called with much higher repeatability and reproducibility than those with lower VF (between 5 and 25%), a level at which many somatic mutations are found. Also, Illumina MiSeq run-to-run reproducibility was significantly higher than  that of Ion Torrent PGM. However, Illumina TruSeq library preparation protocol resulted in much lower repeatability than those obtained from Ion Torrent AmpliSeq protocol at the low VF range. To determine the optimal variant call settings, we used different sets of more stringent filters (lower false positive rate, but higher false negative rate), each platform could achieve close to 95% reproducibility and repeatability. Sequenom MassArray was used as a tie-breaker assay for discordant calls between the two NGS platforms to establish the “truth”. Our data provide insight into the steps that contribute most to variability, such as the procedure of library preparation, the sequencing- by-synthesis chemistry, the factors that impact mutation calls and sampling variation. We also found very high C to T mutation calls associated with Illumina Cancer Panel using TruSeq library preparation protocol (but not with Ion Torrent using AmpliSeq protocol) when a less stringent filter set was used. The C to T artifact mutation calls from formalin-fixed paraffin-embedded (FFPE) tissue samples observed in this study together with high C to A artifact mutation calls from acoustic shearing of intact DNA observed by others have the potential to negatively impact mutation profiling and mutation signature identification if not carefully addressed. Based on these results  we conclude library preparation protocols that start with PCR amplification, such as the AmpliSeq protocol, provide higher repeatability on variant calls with low VF and therefore, are more suitable for mutation profiling and mutation signature studies where somatic mutations (including unknown mutations) are the focus and balanced false positive and false negative rates are critical to success.