Deconstructing FastText

This presentation offers a rigorous and visually structured exploration of how FastText advances natural language processing beyond traditional word-level models. The work begins by critically examining the limitations of early NLP approaches such as Word2Vec and GloVe, which treat words as indivisible atomic units. As illustrated in the early slides, this assumption leads to key challenges, including out-of-vocabulary failures and morphological blindness, in which related word forms (e.g., “walk,” “walking,” “walked”) are learned independently, without shared structure. The presentation further highlights the sparsity crisis in morphologically rich languages, where vocabulary explosion demands excessive data and computational resources. The core contribution of the presentation lies in its detailed exposition of the FastText paradigm. It introduces a compositional representation of words via character n-grams, supported by deterministic tokenisation with boundary markers and sliding-window extraction. The diagrams effectively demonstrate how words are decomposed into overlapping subword units, enabling shared statistical strength across related terms. The mathematical foundation is clearly articulated through the embedding formulation, where a word vector is computed as the average of its subword vectors. The forward pass, training loop, and dataset generation process are presented with both theoretical clarity and practical PyTorch implementation, bridging the gap between concept and code. The presentation also demonstrates applied capabilities, including next-word prediction using a literary corpus and sentiment analysis using supervised FastText models. Evaluation metrics and semantic geometry visualisations provide insight into model performance and interpretability. Finally, the work situates FastText within the broader evolution of NLP, acknowledging its limitation as a static embedding model and positioning contextual embeddings (e.g., ELMo) as the next frontier. Overall, this presentation delivers a comprehensive, research-oriented synthesis of subword-based semantic modelling.