Revisiting Graphones for Language Modelling

Conventionally, text-based vector-symbolic codes have used letters or bytes as their atomic units. We propose using the graphone instead: a unit we adapt from grapheme-to-phoneme conversion that pairs a letter with its corresponding sound. We find that graphones outperform letters and bytes as they let the model utilise the individual streams to build a signal stronger than its constituent parts. We first benchmark graphones against bytes and letters on prediction, grammatical judgement and part-of-speech (POS) selectivity, finding reductions of 0.034 to 0.037 bits per byte that remain significant after Holm correction. We then investigate whether these gains come from coupling each letter and its sound into a single unit or from providing them as separate streams, finding the separate streams to be sufficient: the model keeps weight on the sound stream even when the joint unit is present; without the joint unit, the gains remain.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-19
DOI
https://doi.org/10.5281/zenodo.24356107
Primary Topic
Natural Language Processing Techniques
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Revisiting Graphones for Language Modelling

Ruadhan Chetham
Zenodo (CERN European Organization for Nuclear Research)
Natural Language Processing Techniques
preprint

Revisiting Graphones for Language Modelling

Ruadhan Chetham
preprint en

Abstract

Conventionally, text-based vector-symbolic codes have used letters or bytes as their atomic units. We propose using the graphone instead: a unit we adapt from grapheme-to-phoneme conversion that pairs a letter with its corresponding sound. We find that graphones outperform letters and bytes as they let the model utilise the individual streams to build a signal stronger than its constituent parts. We first benchmark graphones against bytes and letters on prediction, grammatical judgement and part-of-speech (POS) selectivity, finding reductions of 0.034 to 0.037 bits per byte that remain significant after Holm correction. We then investigate whether these gains come from coupling each letter and its sound into a single unit or from providing them as separate streams, finding the separate streams to be sufficient: the model keeps weight on the sound stream even when the joint unit is present; without the joint unit, the gains remain.

Zenodo (CERN European Organization for Nuclear Research)
Natural Language Processing Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon