AudioInSpace: Exploring the Creative Fusion of Generative Audio, Visuals and Gameplay

Abstract

Abstract. Computer games are unique creativity domains in that they elegantly fuse several facets of creative work including visuals, narrative, music, architecture and design. While the exploration of possibilities across facets of creativity offers a more realistic approach to the game design process, most existing autonomous (or semi-autonomous) game content generators focus on the mere generation of single domains (creativity facets) in games. Motivated by the sparse literature on multifaceted game content generation, this paper introduces a multifaceted procedural content generation (PCG) approach that is based on the interactive evolution of multiple artificial neural networks that orchestrate the generation of visuals, audio and gameplay. The approach is evaluated on a spaceship shooter game. The generated artifacts -a fusion of audiovisual and gameplay elements -showcase the capacity of multifaceted PCG and its evident potential for computational game creativity.

Orchestrating Game Generation

“…Interactive evolution has also allowed human intervention on orchestration between multiple game facets, such as space shooter weapons' visuals and trajectories (game rules) in Galactic Arms Race [81]. AudioInSpace [82] requires the player to test the weapons and select which weapons' visuals and rules are best orchestrated with the game's generated soundscape; the player can select which weapon they prefer but also which audio they prefer, allowing for more direct control over which facet should be adjusted to become more harmonious.…”

“…As with bottom-up and top-down approaches, there are different degrees of human interaction, which can be included in the orchestration process. Human interaction can use direct selection as in [78], [79], and [82] to replace the coherence evaluation of Fig. 5, indirectly learned designer preference models, which can automate evaluation until new data are offered by the player [44], [83], [84], or human-provided gameplay data to replace automated playtesting for the gameplay facet [80], [81].…”

“…Audio affects the weapon's behavior; a fired particle, its speed, and color can affect the audio pitch or volume. In-game screenshot is from [82]. generated level places the lair of Baba Yaga close to the king's palace, but the treasure cave is accessed from the palace only by crossing the lair.…”

“…AudioInSpace [82] is a space shooter similar to R-type (Irem 1987), where the game's soundtrack affects the behavior of the player's weapons and vice versa. The weapon's bullets are represented as particles, the position and color of which are controlled by a compositional pattern producing network (CPPN) [92] that uses the game audio's current pitch information and the position of the bullet (relative to where it was fired from) as its input.…”

Interactively Evolving Compositional Sound Synthesis Networks

“…Instead of relying on fitness functions designed by developers, users make their own aesthetic decisions by rating candidate individuals. With IEC users can evolve music and sounds [11,3,15], images [26], and visuals for video games [24,8,10,23]. For example in MutaSynth [3], timbres are encoded as arrays of floats where each gene represents a synthesizer parameter.…”