NCI-MoE: Selective Expert Loading and Concept-Guided Analysis for Mixture-of-Experts Models

Mixture-of-Experts (MoE) models scale to hundreds of billions of parameters but require loading all experts into memory, limiting deployment. We demonstrate that only 6 of 256 experts (2.3%) are needed per inference step, enabling selective loading with 97.7% weight reduction. Using Neural Concept Integration, we analyze DeepSeek V4 Flash (154.5B params, 43 layers) and show concept birth varies by prompt type (L6-L21). Expert usage ranges 156-253 unique experts per prompt, suggesting pruning potential. Memory reduces from 58GB (FP8) to 12-25GB. Experiments run on DGX Spark with patched CUDA kernels for compatibility.