Audio Analyzer

Everything here is ordinary math on the numbers that make up the sound. There's no model trained on music, and nothing leaves your browser. The code is one file, engine.js. It runs in a Web Worker, a background thread, so the page stays smooth while it works. The steps below follow the code in the order it runs. Each one names the function that does it, defines the real terms, and says where you can see the result on this page. The timings assume 44,100 samples a second.

As the sound comes in Stream.push()

1. 1

   Sound in

   Sound reaches the page as samples, 44,100 or 48,000 numbers a second. From the mic, a small AudioWorklet hands them over 1,024 at a time. From a file, the browser decodes the MP3, WAV or other format into the same kind of list. Left and right are averaged into one channel for every step except loudness and stereo.

   capture worklet Stream.push()

   Sample

   One number between −1 and 1: how far the speaker cone (or the mic's diaphragm) is pushed at one instant.

   Sample rate

   Samples per second. 44,100 catches sound up to about 22 kHz, past the top of human hearing.

   AudioWorklet

   A small program that runs on the browser's own audio thread, so no sound gets dropped.

2. 2

   Loudness, sample by sample

   Each sample goes through the K-weighting filter, made of two biquads. The filtered samples are squared and added up in 100 ms blocks. Four blocks make the momentary loudness (400 ms), and thirty make the short-term loudness (3 s). For the true peak, it works out the wave between the samples at 4 times the rate. It only does this near the loudest moment so far, the only place a new peak could be. It also tracks how alike left and right are, for the stereo readings.

   Stream._loudness() _kblocks() _closeBlock()

   On the page: the loudness meter, and the purple line along the bottom of the timeline.

   Biquad

   The smallest standard digital filter: each output mixes this sample and the two before it.

   K-weighting

   A gentle lift above about 1.7 kHz and a cut below about 38 Hz, roughly how a head and ears hear. It comes from the broadcast standard ITU-R BS.1770.

   True peak

   The top of the smooth wave a speaker actually plays. It can fall between two samples and be higher than both.

3. 3

   Short slices, every 11.6 ms

   The samples are cut into overlapping frames of 2,048 samples (46 ms), a new one every 512 samples (11.6 ms, the hop). Each frame is faded in and out with a Hann window. Then an FFT turns it into a spectrum, which is pooled into 96 bands spaced evenly in pitch, from 30 Hz to 16 kHz, the way piano keys are.

   Stream._onset() rfft()

   On the page: the spectrum curve on the dark screen, and the waterfall under it, which is these slices laid side by side.

   FFT

   Fast Fourier Transform: the classic recipe for finding how much of each frequency a slice of wave holds.

   Spectrum

   That answer: strength at each frequency, from low to high.

   Hann window

   A smooth fade at both ends of the slice, so cutting it out doesn't add a click of its own.

4. 4

   Onsets: where sounds start

   For each band, it asks how much louder this frame is than the one before, and counts only the increases. Each band is also compared with its neighbours, so a singer's vibrato sliding between bands doesn't count. The total over all bands is the spectral flux: one number per frame that jumps whenever a drum hits or a note begins. The same total below 150 Hz is kept on its own, to track the kick and the bass.

   Stream._onset() flux, lowFlux

   On the page: the bright vertical streaks in the waterfall.

   Onset

   The moment a sound starts.

   Spectral flux

   How much new energy arrived since the last frame. The neighbour check follows SuperFlux (Böck and Widmer, 2013).

5. 5

   The pitch of one voice, every 23 ms

   Every other frame, YIN takes 23 ms of the wave and compares it with a copy of itself, slid later by every amount from 1/2,200 to 1/43 of a second. A voice or a string repeats its shape once per period, so the difference dips at that slide. The first deep dip wins, and 1 ÷ period is the pitch in hertz. The depth of the dip is the confidence. One FFT does the comparison for every slide at once, which is what makes this fast enough to run live.

   Stream._yin()

   On the page: the pitch meter and its cents bar.

   Period

   How long a wave takes to repeat. A4 repeats 440 times a second, so 440 Hz.

   YIN

   A pitch finder by de Cheveigné and Kawahara (2002): the difference between the wave and its shifted copy, scaled so the first true dip stands out.

6. 6

   Long slices, every 46 ms: finding notes

   Notes need finer detail, so a second set of frames is 8,192 samples long (186 ms). Its frequency points sit about 5 Hz apart, and fitting a small curve (a parabola) through each peak pins a note to a fraction of that. It keeps the 48 strongest peaks between 40 Hz and 5 kHz. Then comes the key idea: a peak that was already there one frame earlier (46 ms), at the same pitch to within 30 cents, belongs to a held note. Drum hits and noise rarely hold still, so they drop out here.

   Stream._long()

   Peak

   A frequency stronger than its neighbours: a note's fundamental, or one of its overtones, the quieter copies at 2, 3, 4 times its frequency that give an instrument its colour.

   Cent

   A hundredth of a semitone, the step between neighbouring piano keys.

   The trade-off

   A longer slice is sharper in pitch but blurrier in time, which is why there are two sets of slices.

7. 7

   Tuning

   Each held peak between 100 Hz and 4 kHz is measured against the nearest piano note, louder peaks counting more. These offsets are averaged around a circle, a circular mean, so that +49 and −49 cents (nearly the same place) don't cancel out to 0. The result is the tuning, where 0 cents means A = 440 Hz.

   Stream._long() tuning()

   On the page: the key card's "Tuned to A = …".

   Circular mean

   An average of positions on a circle, like averaging times on a clock face: 11:59 and 12:01 average to 12:00, not 6:00.

8. 8

   The pitch that stands out in a mix

   In a full band, YIN can't follow one part. So on each long frame, every peak votes for each note it could be the 1st to 20th overtone of, with higher overtones casting weaker votes. This is harmonic summation. The votes are weighted by how sensitive ears are at that frequency (A-weighting). The note with the most votes is the one that stands out, usually the melody.

   Stream._long() melody salience

   On the page: the teal dots on the timeline's spectrogram, and "strongest in the mix" on the pitch meter.

   A-weighting

   A standard curve for how much quieter very low and very high sounds seem to us.

When the take ends Stream.finish()

1. 9

   Notes into 12 pitch classes

   Each held peak becomes its nearest piano key, after correcting for the tuning. Then every octave folds together into the 12 pitch classes, the chroma. Two sets are kept: notes from A2 up, for the harmony, and notes from E1 to D4, where a bass line lives.

   semis() foldChroma()

   On the page: the 12 bars in the chord meter.

   Pitch class

   A note name without its octave: every C on the piano is pitch class C.

   Chroma

   The 12 pitch-class strengths as one list, C through B.

2. 10

   Tempo

   First the onset line from step 4 is cleaned up: subtracting a one-second moving average leaves just the spikes. Then comes autocorrelation, sliding the line along itself to see how often spikes land on spikes. A slide of one beat lines up best. This is done in 8-second windows that are then averaged, so a song that drifts a little still gives one clear answer.

   Every tempo from 40 to 240 bpm gets a score: how well its one-beat slide lines up, plus half the score at two beats. That score is multiplied by a prior, a gentle preference for tempos near 115 bpm, because half and double the tempo fit the spikes almost as well. The winner is refined to a fraction of a bpm. The close alternatives (half, double, 2/3 and 3/2) are kept with their scores, which is how the tempo card knows when to suggest Double.

   prepOnset() onsetACF() tempoFromOnset()

   Autocorrelation

   Comparing a signal with a delayed copy of itself, at every delay. Repeats show up as high scores.

   bpm

   Beats per minute.

   Prior

   A preference set in advance, used only to settle near-ties.

3. 11

   Beats

   With the tempo known, it finds the actual beats by dynamic programming, following Ellis (2007). For every frame, the best chain of beats that could end there is that frame's onset strength plus the best chain ending one beat earlier, minus a penalty when the gap strays from the beat length. Walking back from the best last beat gives every beat at once. Each beat is then nudged onto the peak of its spike, and beats in a silent intro or ending are dropped. A straight line through the beat times (linear regression) gives the average tempo, and how much the tempo varies from bar to bar gives the drift.

   trackBeats() trimBeats() summarizeTempo()

   On the page: the yellow ticks under the waveform. "Click on beats" plays them.

   Dynamic programming

   Solving a huge search by building on the best partial answers, so finding the best path through millions of possibilities takes one pass.

   Linear regression

   The straight line that fits a set of points most closely.

4. 12

   One reading per beat

   The take is cut at the beats, or into quarter seconds if there's no steady beat, and each piece's pitch classes are averaged. It also notes how much held pitch each piece has compared with the take as a whole. A drums-only bar has very little, so later it reads as "no chord".

   finish() segments

   Segment

   One of these pieces: the span from one beat to the next.

5. 13

   Key

   The pitch-class balance of the whole take is compared with a key profile for each of the 24 keys, 12 major and 12 minor, using correlation. Two published profiles are blended: one from listeners' ratings (Krumhansl and Kessler, 1982) and one from counting notes in printed music (Temperley, 1999). The bass line counts at half weight.

   keyFromChroma()

   Key profile

   12 numbers: how typical each note is in a key. In C major, C, E and G score high and C♯ scores low.

   Correlation

   How closely two lists of numbers rise and fall together, from −1 to 1.

6. 14

   Bars and the downbeat

   At each beat it measures two things. One is how much the harmony changes, comparing the two beats before with the two beats after, so a melody that moves every beat averages out. The other is how hard the low end hits. Then it tries bars of 3 and of 4 beats, starting on each possible beat, and keeps the fit where changes and kicks land on beat 1 most often. This is the reading most worth checking.

   findMeter()

   On the page: the bar lines and bar numbers on the timeline. "It's in 3" and "Bar starts one beat later" correct it.

   Downbeat

   Beat 1 of a bar, where chords usually change and the kick usually lands.

   Meter

   How many beats make a bar: 4 for most pop, 3 for a waltz.

7. 15

   Chords

   Each beat's pitch classes are scored against 84 chord templates (major, minor, 7, maj7, m7, diminished and sus4 on all 12 roots), plus "no chord". A template includes each chord note's first few overtones, because a real C also sounds a fainter C and G above it. The score is the cosine similarity, plus credit when the bass plays the root, a little extra for chords that belong to the key, and a small cost for the rarer kinds of chord.

   Then a Viterbi search finds the likeliest run of chords across the whole take. Changing chord has a cost, lowest on beat 1, so it won't flicker on a passing note. A chord note that's clearly strongest in the bass makes a slash chord, such as G/B.

   emissions() decodeChords()

   On the page: the chord lane at the top of the timeline. "Play its chords" sounds them.

   Template

   The pattern of pitch classes a chord should make.

   Cosine similarity

   How alike two lists are in shape, ignoring overall loudness. 1 means the same shape.

   Viterbi

   A way to find the best path through a grid of choices in one pass, keeping only the best route to each choice at each step (Viterbi, 1967). Speech recognition has long used it.

8. 16

   The key, a second look

   The chords now vote on close calls, such as C major against A minor, which share every note. The key wins whose home chord lasts longest, opens and closes the take, and has the most chords that fit. If that changes the key, the chords are read again. In takes longer than a minute, it also looks for key changes, using 8-second windows and a high price on changing.

   refineKey() keySegments()

   On the page: the key card and its circle of fifths.

   Relative keys

   A major key and the minor key that shares all its notes, such as C major and A minor. They're the most common mix-up.

9. 17

   The chart

   Each bar gets one chord, or two if the chord changes halfway through. The chords are written as Nashville numbers, four bars to a line. Lines that repeat become sections A, B and C. If a major chord sits where the key expects a minor one, the chart says the backline page needs its strict reading.

   buildChart() nashville()

   On the page: the chart in the chords card, ready to copy.

   Nashville numbers

   Chords named by their step in the key. In G, G is 1, C is 4, D is 5 and Em is 6m, so the same chart works in any key.

10. 18

    Summing up: loudness, range and tone

    Integrated loudness averages the 400 ms blocks. It skips any block under −70 LUFS, then any more than 10 LU under that first average (this is gating), so pauses don't pull it down. Loudness range is the spread of the 3-second readings, from the 10th to the 95th percentile. If more than about a third of the sound had one clear pitch, it treats the take as one voice and reports its range and tuning. Tone balance compares each octave's energy with pink noise, and the spectral centroid is the spectrum's balance point.

    loudnessSummary() voiceSummary() toneSummary() stereoSummary()

    On the page: the loudness, tone balance and pitch cards.

    LUFS

    Loudness Units relative to Full Scale. Streaming services play music at about −14. An LU is a step of the same size.

    Percentile

    The value a given share of readings fall under. The 95th percentile is louder than 95% of them.

    Pink noise

    Hiss with the same energy in every octave, the usual "flat" reference for tone.

11. 19

    Your corrections

    Half, Double, Tap the beat, It's in 3 and Bar starts one beat later don't listen again. They hand the slices it already measured back to finish() with a hint (a tempo, a meter or a shift), and steps 10 to 17 run again in a moment. While you listen, the live meters use the same slices: the tempo comes from the last 12 seconds, the chord from pitch classes that fade within about a quarter second, and the key from ones that fade over about 40 seconds.

    finish(hints) liveTempo() liveChord() liveKeyNow()

    Hint

    A fact you supply that the math then takes as given.