You know that feeling when you come up with a melody, but forget it moments later? A classic trick is to record yourself humming the idea and build on the concept when you have time. But what if you could turn that poorly hummed melody into a realistic audio sample of a flute or violin performance?
Tone Transfer is an online music app powered by machine learning and designed by Google’s Magenta Team. The app lets users upload audio of a single voice or instrument, then play it back through some other musical instrument. Users can download these sound samples as wav files and import them to any DAW.
On January 26 2023, Google released a paper showcasing a new AI music product called MusicLM that turns text prompts into music.
This article covers the basics of how to use Tone Transfer (web) and the DDSP VST (plugin). I’ll also include some resources at the end for more information about how the neural networks work to make this app possible.
If you're a music producer and want to work directly in the DAW, check these audio-to-midi plugins instead.
Article Contents
Tone Transfer vs. DDSP Plugin
In the video above, Andrew Huang shows off Timbre Transfer, an early Tone Transfer prototype that loaded in Google Colab.
Tone Transfer debuted in October 2020 on Tensorflow. Site visitors didn't need to log in or create an account to begin using the app. It’s a convenient tool that anyone can use, regardless of whether they own music software. And it's free!
On June 8th 2022, the Magenta Team announced the launch of DDSP-VST, a better and more powerful version of Tone Transfer. Also free to download and use.
The DDSP plugin loads on your DAW tracks. You can pass MIDI or raw audio data through it and transform the signal into the instrument of your choice. If you're performing live with the plugin turned on, the DAW will play back the audio output in real time and can even record it.
We'll cover both of these apps in the remainder of this article, starting with Tone Transfer. Then we'll show you how to install and use the DDSP plugin.
How to Use Tone Transfer on the Web
To get started, visit the Tone Transfer website and proceed past the intro screen to the app. Heads up - you can open the dashboard on a desktop, laptop or android browser, but iOS mobile devices are not currently supported.
Tone Transfer has one primary dashboard. Here, you'll be able to select from existing audio inputs or record your own. For the output, a few musical instruments are available; saxophone, flute, violin, and trumpet.
Once you find an audio sample you like, use the download icon to save it to your hard drive. From here, drag it into your DAW and use it as a royalty-free sample.
If you want to dig into the app a bit deeper, pop open the ML model control panel and adjust the output. Access it via the icon in the bottom left corner of the dashboard.
The Mix feature lets you blend the original input with your new instrumental output. You can use the Octave slider to find a better register for the instrument’s timbre, and the Loudness control adjusts for ambient noise in the output.
How to Install DDSP-VST (Tone Transfer Plugin)
Once you’ve tried out Tone Transfer, head on over to the DDSP-VST page and download the free plugin. You’ll have access to all the same Tone Transfer features, but with even more instruments to choose from. Plus you can do some nifty stuff like shape the tone with a cool interface, modify noise levels, add reverb, and more.
The DDSP zip file doesn’t include a standalone application, so you won’t be running a traditional installer. Instead, you’ll just need to drag the VST3 or Component files into the appropriate folder.
Installation instructions
To get started, close your digital audio workstation.
Mac users with Logic Pro should use the component file and load it as an Audio Unit. Otherwise, the rest of the DAWs seem to work well with the VST3 format.
The two most common paths for Mac plugins are:
Library > Audio > Plug-Ins > Components
Library > Audio > Plug-Ins > VST3
The most common path for Windows VST3 plugins:
C:\Program Files (x86)\Common Files\VST3.
If you still need help, see these video tutorials on our FAQ page. We include Logic Pro, Ableton, and FL Studio instructions for Mac and PC.
How to use the DDSP plugin
Once the VST3 or component file located in the right folder, you can open your DAW and load the DDSP plugin onto a track. From here you have two choices - transform a performance in realtime or playback audio as a new instrument.
If you recall, Tone Transfer only took audio recordings for input. DDSP is different.
You can load the DDSP plugin onto an audio or MIDI track and it will work in both scenarios. Audition melodies across different instruments and fine tune the timbre until it's just right. Then use these audio samples to create new music.
The most exciting, cutting edge part of this plugin, in my opinion, is the option to train your very own DDSP models in Google Colab. This may sound like an advanced feature, but it’s not all that difficult in practice. As you train new models, you’ll be able to use add those sounds to your musical instrument collection within the app.
Key Features
Model selection - Select an instrument for your output. You can train a new model in Google Colab notebook and then open a folder to access custom models.
Tone shaping - The instrument’s ideal range is indicated by DDSP’s tone box - that purple rectangle in the screenshot above. It changes with each instrument model. Drag the shaded part of the box in any direction to adjust its timbre. This is surprisingly fun to play with.
Input / Output controls - Choose from a variety of controls including pitch shifting by semitones, harmonic tone presence, instrument noise, and output gain.
Reverb Effects - Adjust the instrument's reverb size, dampness, and wetness.
Train your own model - Upload a 10-20 minute recording of an instrument, like a cello, playing one note at a time. Magenta will use it to generate a custom model that you can load back into the DDSP plugin and run audio through.
Deep Learning with Magenta DDSP
If you find Tone Transfer or DDSP useful, I encourage you to explore how the underlying technology works. It’s a fun entry point to the greater topic of machine learning.
Magenta calls their technique Differentiable Digital Signal Processing, combining classical DSP with machine learning models to recreate everyday sounds.
DDSP doesn't rely on the same techniques that classical additive synthesizers did. The neural network spends a number of hours of training on high quality sounds to produce its model. With the Google Colab, you can try creating your very own models.
Explore the app's open source code repository on Github and join their Discord channel to participate in this growing AI music community. You can also check out our recent article on Magenta Studio, a MIDI generation project from the same team. We provide a broad overview of other popular AI music apps on the market.