Trajectory-based Handwritting Recognition via Spatiotemporal Convolution on Distance Matrix

In this paper, the handwriting recognition task is seen as a trajectory classification problem.Compared to the 2D image of a digit, a trajectory is a highly efficient representation since only a small fraction of the trajectory's points are used.When identifying handwriting styles, the temporal order of trajectory points is crucial.However, in semantic recognition tasks, where the goal is to understand the semantic meaning conveyed by the handwriting characters, the trajectory's shape should dictate its classification, regardless of the writing order.Therefore, existing trajectory classification methods like long short-term memory recurrent neural network (LSTM) or dynamic time warping (DTW) heavily rely on temporal information, which may not be ideal for semantic recognition tasks, where shape is the key determinant of trajectory class.The model proposed in this paper addresses this issue by extracting features of shape similarity in different combinations of sub-trajectories from the distance matrix of paired trajectories.We propose STKNet, a CNN model with special convolution kernels that can extract spatial and temporal features from the distance matrix of two trajectories efficiently.We use STKNet to compute the distances between handwritten trajectories, and then complete the classification using the -nearest neighbor method, calling the whole process STKNet+NN.We evaluate our model on the MNIST stroke sequence dataset, which consists of trajectory data recovered from the original MNIST image data.Our experiment shows that training on trajectories as inputs can achieve near state-of-the-art prediction accuracy with only 1,200 trajectories in the training sample, a relatively small number of labeled samples compared to 60,000 MNIST training images.This can be attributed to the richer information contained in trajectories.The experiment also shows that our method still has high accuracy in recognizing digits with many different writing styles.