Text to speech

Supertonic 2 turns text into speech in four stages: it reads the characters, predicts how long the speech will be, refines random noise into a compact description of the sound in a few steps, and decodes that into a waveform. The spectrogram and pitch line are computed from its output with the code from the spectrogram lab.

Try this

Switch voices and watch the orange pitch line and the spacing of the harmonic lines move together.
Compare 1 and 10 denoising steps for the same sentence and voice.
Change Starting noise and listen for small differences in timing and tone.
Speed the voice up to 2x and see what Whisper makes of it.

Why the number of steps matters, in NumPyPython with NumPy

import numpy as np

# A toy flow-matching sampler. The "data" are two possible values, -1 and +1
# (think: two ways to say a sound). Samples start as Gaussian noise at t = 0 and
# follow a velocity field to t = 1. For this data the ideal velocity is exact:
# v(x, t) = (E[data | x at time t] - x) / (1 - t), with E[...] = tanh(x t / (1 - t)^2).
def velocity(x, t):
    return (np.tanh(x * t / (1 - t) ** 2) - x) / (1 - t)

rng = np.random.default_rng(0)
start = rng.standard_normal(10_000)
for steps in (1, 2, 5, 10, 50):
    x = start.copy()
    for k in range(steps):                     # Euler steps from t = 0 to t = 1
        t = k / steps
        x = x + velocity(x, t) / steps
    near = np.mean(np.abs(np.abs(x) - 1) < 0.1)
    print(f"{steps:2d} steps: mean |x| = {np.mean(np.abs(x)):.3f}, within 0.1 of -1 or +1: {near:.0%}")

 1 steps: mean |x| = 0.000, within 0.1 of -1 or +1: 0%
 2 steps: mean |x| = 0.559, within 0.1 of -1 or +1: 14%
 5 steps: mean |x| = 0.986, within 0.1 of -1 or +1: 97%
10 steps: mean |x| = 1.000, within 0.1 of -1 or +1: 100%
50 steps: mean |x| = 1.000, within 0.1 of -1 or +1: 100%