Deep neural networks and stochastic methods for cognitive modeling of rat behavioral dynamics in $$\mathbb {T}$$-mazes

Modeling animal decision-making requires mathematical rigor and computational analysis to capture underlying cognitive mechanisms. This study presents a cognitive model for rat decision-making behavior in <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mi>T</mml:mi></mml:math> -mazes by combining stochastic methods with deep neural architectures. The model adapts Wyckoff's stochastic framework, originally grounded in Bush's discrimination learning theory, to describe probabilistic transitions between directional choices under reinforcement contingencies. The existence and uniqueness of solutions are demonstrated via fixed-point theorems, ensuring the formulation is well-posed. The asymptotic properties of the system are examined under boundary conditions to understand the convergence behavior of decision probabilities across trials. Empirical validation is performed using Monte Carlo simulations to compare expected trajectories with the model's predictive output. The dataset comprises spatial trajectory recordings of rats navigating toward food rewards under controlled experimental protocols. Trajectories are preprocessed through statistical filtering, augmented to address data imbalance, and embedded using t-SNE to visualize separability across behavioral states. A hybrid convolutional-recurrent neural network (CNN-LSTM) is trained on these representations and achieves a classification accuracy of 82.24%, outperforming conventional machine learning models, including support vector machines and random forests. In addition to discrete choice prediction, the network reconstructs continuous paths, enabling full behavioral sequence modeling from partial observations. The integration of stochastic dynamics and deep learning develops a computational basis for analyzing spatial decision-making in animal behavior. The proposed approach contributes to computational models of cognition by linking observable behavior to internal processes in navigational tasks.