Hyper-Exponential Evolution of AI Systems

Ray Kurzweil argues that by 2045, the Singularity will arrive, at which point the computing power of computerswill surpass that of humanity. His claim is based on Moore’s Law, which states that technological advancementsincrease chip density, thereby shortening signal transmission distances and increasing transmission speeds, ultimatelyresulting in faster computations. However, this argument does not account for the programs written on these chips.It assumes that if program size remains fixed while chip density increases, computations will automatically becomefaster. Now, suppose the physical size of the chip remains unchanged. When the density increases, the size of theprograms it can store increases. Furthermore, a program that is N times larger can handle more than N times theamount of information. If computations rely solely on local communication on the chip, the speed increases by a factorof !N, while the computational capacity becomes N times larger. If computations require global communication, onthe other hand, the speed remains unchanged, aligning with Kurzweil’s prediction. In the case of vector computations,such as those used in GPUs for Transformers operating on SIMD (Single Instruction, Multiple Data) architectures,computation is localized. This implies that not only does the speed increase, but the complexity of the program mustalso increase. The central argument of this paper is that the overall improvement in speed would not merely be exponential,as Kurzweil claims, but rather exponential of the exponential. Although we do not explore this further, our argumentshould apply to a broader range of parallel computing architectures.