Predicting the Influence of Input Structure, Sequence, and Architecture on Neural Network Representations

This research investigates how three rarely unified factors—the internal structure of the data, the sequence in which inputs are presented, and the inductive biases of a neural network's architecture—jointly determine the range of representations a model is likely to learn during training. Rather than assuming neural networks converge to a single predictable solution, this work demonstrates that different combinations of these factors guide models into distinct, stable “representational basins.” To evaluate this hypothesis, we constructed controlled synthetic datasets with configurable structure, multiple curriculum-like input sequences, and three different neural architectures (MLP, CNN, Transformer). Each condition was independently trained, and the resulting hidden-layer representations were analyzed using PCA visualizations, CKA similarity, cosine similarity, and variance correlation metrics. A shared test dataset ensures that comparisons reflect true differences in learned internal geometry, rather than superficial sampling variance. The findings show clear, measurable evidence that: Data structure strongly shapes the geometry of learned feature spaces, Input ordering meaningfully anchors early gradient directions, shifting representational outcomes, Architectural bias interacts with both of the above to define the representational range accessible to a model. The accompanying codebase runs all 27 experimental conditions automatically, saves PCA plots, computes all similarity measures, and exports a full CSV of results. This makes the research fully reproducible and transparent. This work contributes a practical framework for understanding and predicting how models internalize data, offering implications for interpretability, curriculum design, and data-centric model development. Files Included: Full research paper (PDF) Complete experiment pipeline (Python) PCA plots of learned representations Representation similarity CSV (CKA, cosine, variance) Synthetic dataset generation code