Revisiting Graphones for Language Modelling
Conventionally, text-based vector-symbolic codes have used letters or bytes as their atomic units. We propose using the graphone instead: a unit we adapt from grapheme-to-phoneme conversion that pairs a letter with its corresponding sound. We find that graphones outperform letters and bytes as they let the model utilise the individual streams to build a signal stronger than its constituent parts. We first benchmark graphones against bytes and letters on prediction, grammatical judgement and part-of-speech (POS) selectivity, finding reductions of 0.034 to 0.037 bits per byte that remain significant after Holm correction. We then investigate whether these gains come from coupling each letter and its sound into a single unit or from providing them as separate streams, finding the separate streams to be sufficient: the model keeps weight on the sound stream even when the joint unit is present; without the joint unit, the gains remain.
Authors
- Ruadhan Chetham (ORCID: https://orcid.org/0009-0009-1305-700X)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-19
- DOI
- https://doi.org/10.5281/zenodo.24356107
- Primary Topic
- Natural Language Processing Techniques
- Type
- preprint