Equation Tokenization for LLM Data Processing

In this work is presented a data processing approach for scientific equation tokenization as a part of the language models predefined stage. This way, equations are considered as a single-line text expressions, that makes possible the usage of 1D function prediction neural networks with reduced complexity. Three scenarios for usage of tokenized equations are considered, they are: next token prediction; probabilistic token aggregation and token appearance correlation with data fitting. These scenarios are inspired by text processing language models. An LSTM neural network example is implemented for next symbol in equation prediction.