Discovering Underlying Pattern Clusters by Encoding Data to Transformer Tokens

The fundamental challenge of underlying pattern discovery remains unresolved in the current era of general artificial intelligence. Inspired by the excellent token sequence representation capabilities of transformer-based large language models, we think that data patterns can be effectively represented through sequences of symbolic tokens. So, this study presents a novel underlying pattern clustering discovery model (UPCD), featuring three groundbreaking innovations: (1) Attribute Data Symbolization Mechanism: A dual-phase autoencoder that transforms raw data into semantically rich token sequences, opening up new horizons for traditional data mining tasks. (2) Data Token Embedding Mechanism: Implementing masked predictive learning to establish dynamic token relationships, enabling pattern-aware representation spaces. (3) Semantic Encoding Aggregation Mechanism: Introducing vector quantization to adaptively cluster data, unearthing its underlying patterns. Comprehensive benchmarks reveal UPCD’s unprecedented performance: achieving 88.39% accuracy (+3.14% over KMCC) and 58.95% Adjusted Rand Index (+8.83% improvement) in NSL-KDD cybersecurity data. Furthermore, real-world validation through unknown TCP traffic analysis demonstrates its exceptional pattern mining capability, outperforming existing adaptive algorithms by 4-21% in the Silhouette Coefficient. By reconstructing raw structured data with token sequences, UPCD pioneers new frontiers for intelligent data mining. The implementation details and code are available at https://github.com/kcisgroup/UPCD.