Brief: Explore how machine learning tools can help artists create art and music in new ways
Solution: An experimental physical interface for the NSynth algorithm
Press: CNBC, The Verge, Fast Company, DJMag, Resident Advisor, Engadget, Sonic State
Technology has always played a role in creating new types of sounds that inspire musicians—from the sounds of distortion to the electronic sounds of synths. Today, advances in machine learning and neural networks have opened up new possibilities for sound generation.
Building upon past research in this field, Magenta created NSynth (Neural Synthesizer). It’s a machine learning algorithm that uses a deep neural network to learn the characteristics of sounds, and then create a completely new sound based on these characteristics.
Rather than combining or blending the sounds, NSynth synthesizes an entirely new sound using the acoustic qualities of the original sounds—so you could get a sound that’s part flute and part sitar all at once.
16 original source sounds across a range of 15 pitches were recorded in a studio and then input into the NSynth algorithm, to precompute the new sounds.
The outputs, over 100,000 new sounds, were then loaded into the experience prototype.
Each dial was assigned 4 source sounds. Using the dials, musicians can select the source sounds they would like to explore between, and drag their finger across the touchscreen to navigate the new sounds which combine the acoustic qualities of the 4 source sounds.
NSynth Super can be played via any MIDI source, like a DAW, sequencer or keyboard.
NSynth is an algorithm that can generate new sounds by combining the features of existing sounds. To do that, the algorithm takes different sounds as input.
Using an autoencoder, it extracts 16 defining temporal features from each input. These features are then interpolated linearly to create new embeddings (mathematical representations of each sound). These new embeddings are then decoded into new sounds, which have the acoustic qualities of both inputs.
NSynth is an algorithm that can generate new sounds by combining the features of existing sounds. To do that, the algorithm takes different sounds as input.
Using an autoencoder, it extracts 16 defining temporal features from each input. These features are then interpolated linearly to create new embeddings (mathematical representations of each sound). These new embeddings are then decoded into new sounds, which have the acoustic qualities of both inputs.
NSynth Super is built using open source libraries, including TensorFlow and openFrameworks, to enable a wider community of artists, coders, and researchers to experiment with machine learning in their creative process.
The open source version of the NSynth Super including all of the source code, schematics, and design templates are available for download on GitHub.
Åsögatan 135
116 24 Stockholm, Sweden
© current state 2020
Phone: +46 761 730 882
Email: contact@currentstate.se