NeRF and Niagara Particles System: Transforming classical music composition into interactive music VR experience

<p xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" class="first" dir="auto" id="d4043933e55">Generative AI has brought the need to redefine the forms of creative expression. The possibilities of AI-driven visual content creation allow contemporary composers to extend their creative abilities and become creators of audio-visual experiences. The application of interactivity, easily achievable in VR environments, further enhances the degree of user immersion, enabling the implementation of a primary music composition idea through visual representation and interaction design. This article presents a new stage in the development of interactive volumetric VR music composition, relying on neural radiance fields (NeRF) and Gaussian Splatting (GS) to build an interactive music VR experience that amplifies the ideological message of an existing classical music composition. This is accomplished by coupling a 3D GS representation with the Niagara particle system. To our knowledge, this is the first attempt to use NeRF and GS in the context of an interactive music VR application.