Semantic Axis Decomposition of Transformer Embeddings

This work introduces a novel method for interpreting sentence-transformer embeddings via semantic axis decomposition, capsule layering, and recursive reasoning. Top-N dimensions are selected using Random Forests and assigned human-interpretable meanings such as “emotionality”, “scientificness”, and “question intent”. We go beyond scalar embeddings by stacking multi-layered capsule sub-coordinates and defining recursive semantic floors. Each floor is traversed using two parameters: alpha (influence magnitude) and theta (semantic shift direction), enabling low-cost, interpretable reasoning paths. A new mechanism—floor pruning—discards semantically weak layers to retain only impactful shifts. Our proposed structure was validated in the Capsule AI Evolution experiment, achieving a +4.45% accuracy gain over baseline without fine-tuning the transformer. This confirms that semantic floor traversal enables dynamic, modular reasoning and performance improvement through capsule-space navigation alone. This is a conceptual and visual demonstration. Code and GUI tools are released separately. Keywords: transformer, embeddings, interpretability, semantic floors, latent space, capsule layers, XAI, sentence-transformers, recursive reasoning Interactive prototype:https://github.com/kexi-bq/embedding-explainer Experiment and evaluation code:https://github.com/kexi-bq/capsule-ai-evolution Capsule Selection In this framework, capsule selection (e.g., dim_5.3) is not fixed in advance but is discovered through an evolutionary optimization process driven by Capsule AI.The system applies random shifts along selected capsule dimensions and evaluates their impact on downstream task performance (e.g., classification accuracy).If a particular capsule configuration improves results, it is rewarded and retained; otherwise, it is discarded. Through multiple iterations, Capsule AI gradually identifies and accumulates effective capsule directions — allowing the embedding space to be optimized without fine-tuning the transformer model itself. Post-Publication Experiment: Capsule Reactivation of Neutral Dimensions We conducted a follow-up experiment targeting neutral embedding dimensions — those that showed <0.001 change in classification accuracy when zeroed out. Task:Multi-class classification of 150 text phrases into 4 categories:emotional, scientific, neutral, question(see: Semantic_Test_Dataset.csv) Baseline setup: Embeddings: all-MiniLM-L6-v2 Classifier: LogisticRegression Train/test split: 70/30 Procedure: Identified 383 “neutral” dims using accuracy-preserving zeroing Applied Capsule Shift (dim += alpha) individually to each dim in test data Measured accuracy impact per shift Results: 87 out of 383 dims (22.7%) produced accuracy improvements Best delta: +2.22% (from 93.33% to 95.56%) This confirms that many “dead” dimensions are semantically reactivatable through capsule-based manipulation — even without retraining the model. For correspondence: Aleksey Schetnikov Email: [email protected] Telegram: @Alex_larinov