Hybrid Spiking Language Model: Combining Spike Counts and Membrane Potentials for Energy-Efficient and Noise-Robust Character Prediction

I propose a novel character-level language model using Spiking Neural Networks (SNNs) that combines both spike counts and membrane potentials for output prediction. This v3 extends the model with BitNet ternary weights and RWKV-inspired time-mixing. Key Findings (v3 NEW):- BitNet Mixed Precision: PPL 2.69 BEATS standard SNN (3.29)!- RWKV Time-Mixing: 36.1% improvement in long-range memory- Ultimate Architecture: 43.4% improvement combining all techniques- Multiplication-free reservoir: 50-70% of operations are additions only- 16-model ensemble achieves PPL 1.04 Key Findings (v1-v2):- SNN achieves BEST perplexity (PPL=9.90) vs DNN (11.28) and LSTM (15.67)- 14.7× more energy-efficient through sparse computation (only 7.6% of neurons fire)- 39.7% quality improvement from hybrid (spike + membrane) approach- Extreme compressibility: 80% neuron pruning and 4-bit quantization still work- 8× memory compression with minimal quality loss- Noise robust: No degradation at 30% input noise This v3 establishes hybrid SNNs with ternary weights as the optimal architecture for ultra-efficient edge AI language processing. Source code: https://github.com/hafufu-stack/snn-language-model