QUANTIFYING INTERSPECIFIC COMPETITION BETWEEN CANCER AND NORMAL CELLS USING USING NONLINEAR MIXED EFFECTS AND ORDINARY DIFFERENTIAL EQUATION MODELING

Tumor growth has been widely studied through various methodologies. In mathematical oncology, researchers use ordinary differential equations (ODEs) to analyze tumor dynamics. These models present meaningful parameters to link mathematical theory with experimental data. For in vitro cocultures, parameters quantifying cellular competition clarify interactions between tumor and normal cells. This research investigates the interaction between cancer and normal cells during competition, focusing on the in vitro growth of SK-MEL-147 (metastatic melanoma) and HaCaT (immortalized epithelial cells) cell lines. Using an ODE model with cell numbers as dependent variables, we quantify interspecific competition through the parameters α_{12} (impact of SK-MEL-147 on HaCaT) and α_{21} (impact of HaCaT on SK-MEL-147). The in vitro cell growth experiments from Morais \textit{et al}., (2017), https://doi.org/10.1038/s41598-017-07553-6, allowed us to estimate parameters for Gatenby's 1996 ODE model. We used a nonlinear mixed effects model from NLMEModeling (https://doi.org/10.48550/arXiv.2011.06879) to account for observation errors and biological variability. The curve fitting matched the experimental data for both cell types. Parameter estimates showed that SK-MEL-147 cells experienced stronger inhibition from HaCaT cells than the reverse, suggesting normal cells hinder cancer cell growth upon contact. Nonlinear mixed effects modeling successfully fit Gatenby's mathematical model to the experimental data, providing competition parameters that clarified interspecific interactions in tumor dynamics. Such models can predict cell growth behavior, supporting experimental design and reducing the need for preliminary \textit{in vitro} tests.