Audio Analyzer

Everything here is ordinary math on the numbers that make up the sound. There's no model trained on music, and nothing leaves your browser. The code is one file, engine.js. It runs in a Web Worker, a background thread, so the page stays smooth while it works. The steps below follow the code in the order it runs. Each one names the function that does it, defines the real terms, and says where you can see the result on this page. The timings assume 44,100 samples a second.

Each step also has a picture of it at work on your sound, at the playhead. Press play and watch them move, drag them, or step one slice at a time with the buttons that follow you down the page. The pictures of one moment run the engine’s own code again right there, and the rest show what it kept along the way. Each one can show you its code. To watch the decisions happen one at a time, in the order the engine makes them, press Step through it.

As the sound comes in Stream.push()

1. 1

   Sound in

   Sound reaches the page as samples, 44,100 or 48,000 numbers a second. From the mic, a small AudioWorklet hands them over 1,024 at a time. From a file, the browser decodes the MP3, WAV or other format into the same kind of list. Left and right are averaged into one channel for every step except loudness and stereo, or, when one channel is wired backwards and the two would cancel, half their difference is taken instead.

   capture worklet Stream.push()

   Sample

   One number between −1 and 1: how far the speaker cone (or the mic's diaphragm) is pushed at one instant.

   Sample rate

   Samples per second. 44,100 catches sound up to about 22 kHz, past the top of human hearing.

   AudioWorklet

   A small program that runs on the browser's own audio thread, so no sound gets dropped.

2. 2

   Loudness, sample by sample

   Each sample goes through the K-weighting filter, made of two biquads. The filtered samples are squared and added up in 100 ms blocks. Four blocks make the momentary loudness (400 ms), and thirty make the short-term loudness (3 s). For the true peak, it works out the wave between the samples at 4 times the rate. It only does this near the loudest moment so far, the only place a new peak could be. It also tracks how alike left and right are, for the stereo readings.

   Stream._loudness() _kblocks() _closeBlock()

   On the page: the loudness meter, and the purple line along the bottom of the timeline.

   Biquad

   The smallest standard digital filter: each output mixes this sample and the two before it.

   K-weighting

   A gentle lift above about 1.7 kHz and a cut below about 38 Hz, roughly how a head and ears hear. It comes from the broadcast standard ITU-R BS.1770.

   True peak

   The top of the smooth wave a speaker actually plays. It can fall between two samples and be higher than both.

3. 3

   Short slices, every 11.6 ms

   The samples are cut into overlapping frames of 2,048 samples (46 ms), a new one every 512 samples (11.6 ms, the hop). Each frame is faded in and out with a Hann window. Then an FFT turns it into a spectrum, which is pooled into 96 bands spaced evenly in pitch, from 30 Hz to 16 kHz, the way piano keys are.

   Stream._onset() rfft()

   On the page: the spectrum curve on the dark screen, and the waterfall under it, which is these slices laid side by side.

   FFT

   Fast Fourier Transform: the classic recipe for finding how much of each frequency a slice of wave holds.

   Spectrum

   That answer: strength at each frequency, from low to high.

   Hann window

   A smooth fade at both ends of the slice, so cutting it out doesn't add a click of its own.

4. 4

   Onsets: where sounds start

   For each band, it asks how much louder this frame is than the one before, and counts only the increases. Each band is also compared with its neighbours, so a singer's vibrato sliding between bands doesn't count. The total over all bands is the spectral flux: one number per frame that jumps whenever a drum hits or a note begins. The same total below 150 Hz is kept on its own, to track the kick and the bass. So are three more, under 400 Hz, from there to 2.5 kHz and over it, for the tempo (step 10). Loudness is counted on a curve that flattens out 60 dB down, and a take far under the usual level (a phone mic in a room can hear music at −50 dB) is lifted by up to 40 dB first, so its quiet onsets still count.

   Stream._onset() flux, lowFlux, bandFlux

   On the page: the bright vertical streaks in the waterfall.

   Onset

   The moment a sound starts.

   Spectral flux

   How much new energy arrived since the last frame. The neighbour check follows SuperFlux (Böck and Widmer, 2013).

5. 5

   The pitch of one voice, every 23 ms

   Every other frame, YIN takes 23 ms of the wave and compares it with a copy of itself, slid later by every amount from 1/2,200 to 1/38 of a second (a bass's low E repeats 41 times a second). A voice or a string repeats its shape once per period, so the difference dips at that slide. The first deep dip wins, and 1 ÷ period is the pitch in hertz. The depth of the dip is the confidence. One FFT does the comparison for every slide at once, which is what makes this fast enough to run live.

   Stream._yin()

   On the page: the pitch meter and its cents bar.

   Period

   How long a wave takes to repeat. A4 repeats 440 times a second, so 440 Hz.

   YIN

   A pitch finder by de Cheveigné and Kawahara (2002): the difference between the wave and its shifted copy, scaled so the first true dip stands out.

6. 6

   Long slices, every 46 ms: finding notes

   Notes need finer detail, so a second set of frames is 8,192 samples long (186 ms). Its frequency points sit about 5 Hz apart, and fitting a small curve (a parabola) through each peak pins a note to a fraction of that. It keeps the 48 strongest peaks between 40 Hz and 5 kHz. Then comes the key idea: a peak that was already there one frame earlier (46 ms), at the same pitch to within 30 cents, belongs to a held note. Drum hits and noise rarely hold still, so they drop out here.

   Stream._long()

   Peak

   A frequency stronger than its neighbours: a note's fundamental, or one of its overtones, the quieter copies at 2, 3, 4 times its frequency that give an instrument its colour.

   Cent

   A hundredth of a semitone, the step between neighbouring piano keys.

   The trade-off

   A longer slice is sharper in pitch but blurrier in time, which is why there are two sets of slices.

7. 7

   Tuning

   Each held peak between 100 Hz and 4 kHz is measured against the nearest piano note, louder peaks counting more. These offsets are averaged around a circle, a circular mean, so that +49 and −49 cents (nearly the same place) don't cancel out to 0. The result is the tuning, where 0 cents means A = 440 Hz. How closely the offsets agree, from 0 to 1, is how sure it is: under 0.6, as with drums alone or noise, the page calls the tuning unclear. One voice goes by its notes instead (step 18), since a wide vibrato swings its peaks with it and their mean can land half a semitone off.

   Stream._long() tuning()

   On the page: the key card's "Tuned to A = …", or "Tuning unclear".

   Circular mean

   An average of positions on a circle, like averaging times on a clock face: 11:59 and 12:01 average to 12:00, not 6:00.

8. 8

   The pitch that stands out in a mix

   In a full band, YIN can't follow one part. So on each long frame, every peak votes for each note it could be the 1st to 20th overtone of, with higher overtones casting weaker votes. This is harmonic summation. The votes are weighted by how sensitive ears are at that frequency (A-weighting). The note with the most votes is the one that stands out, usually the melody.

   Stream._long() melody salience

   On the page: the teal dots on the timeline's spectrogram, and "strongest in the mix" on the pitch meter.

   A-weighting

   A standard curve for how much quieter very low and very high sounds seem to us.

When the take ends Stream.finish()

1. 9

   Notes into 12 pitch classes

   Each held peak becomes its nearest piano key, after correcting for the tuning. Then every octave folds together into the 12 pitch classes, the chroma. Two sets are kept: notes from A2 up, for the harmony, and notes from E1 to D4, where a bass line lives.

   semis() foldChroma()

   On the page: the 12 bars in the chord meter.

   Pitch class

   A note name without its octave: every C on the piano is pitch class C.

   Chroma

   The 12 pitch-class strengths as one list, C through B.

2. 10

   Tempo

   First the onset line from step 4 is cleaned up: subtracting a one-second moving average leaves just the spikes. Then comes autocorrelation, sliding the line along itself to see how often spikes land on spikes. A slide of one beat lines up best. This is done in 8-second windows that are then averaged, so a song that drifts a little still gives one clear answer.

   Every tempo from 40 to 240 bpm gets a score: how well its one-beat slide lines up, plus half the score at two beats. That score is multiplied by a prior, a gentle preference for tempos near 115 bpm, because half and double the tempo fit the spikes almost as well. Then the winner, half it and twice it are scored once more with the lows, the middle and the top each heard apart (step 4's three totals, each made an onset line of its own), averaged with the whole mix: on the whole mix the hi-hat's quick ticks count the most, and in a slow ballad they can outscore the beat itself. The winner is refined to a fraction of a bpm. The close alternatives (half, double, 2/3 and 3/2) are kept with their scores, which is how the tempo card knows when to suggest Double.

   prepOnset() onsetACF() tempoFromOnset()

   Autocorrelation

   Comparing a signal with a delayed copy of itself, at every delay. Repeats show up as high scores.

   bpm

   Beats per minute.

   Prior

   A preference set in advance, used only to settle near-ties.

3. 11

   Beats

   With the tempo known, it finds the actual beats by dynamic programming, following Ellis (2007). For every frame, the best chain of beats that could end there is that frame's onset strength plus the best chain ending one beat earlier, minus a penalty when the gap strays from the beat length. Walking back from the best last beat gives every beat at once. Each beat is then nudged onto the peak of its spike, and beats in a silent intro or ending are dropped. A straight line through the beat times (linear regression) gives the average tempo, and how much the tempo varies from bar to bar gives the drift.

   trackBeats() trimBeats() summarizeTempo()

   On the page: the yellow ticks under the waveform. "Click on beats" plays them.

   Dynamic programming

   Solving a huge search by building on the best partial answers, so finding the best path through millions of possibilities takes one pass.

   Linear regression

   The straight line that fits a set of points most closely.

4. 12

   One reading per beat

   The take is cut at the beats, or into quarter seconds if there's no steady beat, and each piece's pitch classes are averaged. It also notes how much held pitch each piece has compared with the take as a whole. A drums-only bar has very little, so later it reads as "no chord".

   finish() segments

   Segment

   One of these pieces: the span from one beat to the next.

5. 13

   Key

   The pitch-class balance of the whole take is compared with a key profile for each of the 24 keys, 12 major and 12 minor, using correlation. Two published profiles are blended: one from listeners' ratings (Krumhansl and Kessler, 1982) and one from counting notes in printed music (Temperley, 1999). The bass line counts at half weight, or less when the take has almost nothing under 200 Hz: a phone's mic hears no bass notes, only their overtones, whose fifths and thirds would pull a minor key toward its relative major.

   keyFromChroma()

   Key profile

   12 numbers: how typical each note is in a key. In C major, C, E and G score high and C♯ scores low.

   Correlation

   How closely two lists of numbers rise and fall together, from −1 to 1.

6. 14

   Bars and the downbeat

   At each beat it measures two things. One is how much the harmony changes, comparing the two beats before with the two beats after, so a melody that moves every beat averages out. The other is how hard the low end hits. Then it tries bars of 3 and of 4 beats, starting on each possible beat, and keeps the fit where changes and kicks land on beat 1 most often. This is the reading most worth checking.

   findMeter()

   On the page: the bar lines and bar numbers on the timeline. "It's in 3" and "Bar starts one beat later" correct it.

   Downbeat

   Beat 1 of a bar, where chords usually change and the kick usually lands.

   Meter

   How many beats make a bar: 4 for most pop, 3 for a waltz.

7. 15

   Chords

   Each beat's pitch classes are scored against 84 chord templates (major, minor, 7, maj7, m7, diminished and sus4 on all 12 roots), plus "no chord". A template includes each chord note's first few overtones, because a real C also sounds a fainter C and G above it. The score is the cosine similarity, plus credit when the bass plays the root, a little extra for chords that belong to the key, and a small cost for the rarer kinds of chord.

   Then a Viterbi search finds the likeliest run of chords across the whole take. Changing chord has a cost, lowest on beat 1, so it won't flicker on a passing note. A chord note that's clearly strongest in the bass makes a slash chord, such as G/B.

   emissions() decodeChords()

   On the page: the chord lane at the top of the timeline. "Play its chords" sounds them.

   Template

   The pattern of pitch classes a chord should make.

   Cosine similarity

   How alike two lists are in shape, ignoring overall loudness. 1 means the same shape.

   Viterbi

   A way to find the best path through a grid of choices in one pass, keeping only the best route to each choice at each step (Viterbi, 1967). Speech recognition has long used it.

8. 16

   The key, a second look

   The chords now vote on close calls, such as C major against A minor, which share every note. The key wins whose home chord lasts longest, opens and closes the take, and has the most chords that fit. If that changes the key, the chords are read again. In takes longer than a minute, it also looks for key changes, using 8-second windows and a high price on changing. A song that changes key is in the key that sounds longest (a part in its relative key counting with it), and each part's chords are numbered from its own key.

   refineKey() keySegments()

   On the page: the key card and its circle of fifths.

   Relative keys

   A major key and the minor key that shares all its notes, such as C major and A minor. They're the most common mix-up.

9. 17

   The chart

   Each bar gets one chord, or two if the chord changes halfway through. The chords are written as Nashville numbers, four bars to a line. Lines that repeat become sections A, B and C. Where the key changes, a new line starts after a key line such as "Key: up 2", as the backline pages write one, and the numbers after it count from the new key. If a major chord sits where the key expects a minor one, the chart says the backline page needs its strict reading.

   buildChart() nashville()

   On the page: the chart in the chords card, ready to copy.

   Nashville numbers

   Chords named by their step in the key. In G, G is 1, C is 4, D is 5 and Em is 6m, so the same chart works in any key.

10. 18

    Summing up: loudness, range and tone

    Integrated loudness averages the 400 ms blocks. It skips any block under −70 LUFS, then any more than 10 LU under that first average (this is gating), so pauses don't pull it down. Loudness range is the spread of the 3-second readings, from the 10th to the 95th percentile. If more than about a third of the sound had one clear pitch, it treats the take as one voice and reports its range and tuning. In a band that clear pitch is often the bass line, so a part that sits mostly under G2 has to be clear nearly all the time (85%). A held chord can fake one pitch too: C, E and G together repeat like a low C that nobody plays. So a moment counts only if the long slice has a peak at that pitch (or, when a phone's mic loses a low note's own pitch, all of its first overtones from 300 Hz up), and most of what it holds sits on that pitch's harmonics. Every level test here is against the take's own level and the noise in its rests, so a quiet take, or a solo with hiss between its phrases, reads the same as a loud clean one. Above the readings, the page says what it heard: a band, one voice, drums only, or only noise or hum (step 19).

    One voice is also written out as notes. The pitch track gets a five-frame median, which clears single-frame octave slips. A new note starts where a clear pitch begins, where the pitch moves to another semitone and stays there, or where the same pitch is played again: an onset, with the level climbing 3 dB or more out of a dip. Frames on a slide between two notes belong to neither, and so do frames with nothing sounding at their pitch: in a room's echo, the note before and the note now repeat together like a low note nobody plays. Anything under 70 ms is dropped. Each note is rounded to the nearest semitone from the notes' own tuning (where their middles sit, which is the take's tuning too), since a wide vibrato pulls the peaks' reading off. With a steady beat, each length is then rounded to the nearest sixteenth. Tone balance compares each octave's energy with pink noise, and the spectral centroid is the spectrum's balance point.

    loudnessSummary() voiceSummary() alone() noteList() toneSummary() stereoSummary()

    On the page: the loudness, tone balance and pitch cards, and for one voice the notes card.

    LUFS

    Loudness Units relative to Full Scale. Streaming services play music at about −14. An LU is a step of the same size.

    Percentile

    The value a given share of readings fall under. The 95th percentile is louder than 95% of them.

    Pink noise

    Hiss with the same energy in every octave, the usual "flat" reference for tone.

11. 19

    Drums alone

    Before the key, it asks whether there are any notes. In each long slice it takes the peaks at 80 Hz and up, and keeps the share of their energy that was already there, at the same pitch, in the three slices before (140 ms). Notes ring on and drum hits die away: on the backline songs a drum kit keeps under 16%, a bass line alone 24% or more, a band 30% or more. Under 19% it hears drums only, so it looks for no key or chords, and finds the bars from the kick alone. A phone's mic hears no kick, which leaves a kit's snare and toms more of the share, so the same share is also taken from 250 Hz up, where a kit keeps under half on any mic and a band three quarters or more. One voice that is clear most of the time is never drums: a singer's vibrato moves its peaks too far from slice to slice to count as held. Hiss holds nothing either, so drums also need a steady pulse or hits that stand out. Without them it hears only noise or hum, and reads no key, chords or notes. One pitch under C2 that never moves or starts again is hum too (mains hum is 50 or 60 Hz), not one voice.

    Then it writes the drums out. First the swing, from the rises above 2 kHz, where hats and snares are: in each beat, where the strongest hit between the beats lands. If a quarter of the beats have a hit a quarter of the way in, the sixteenths swing; if not, the eighths. Then each sixteenth gets how much every band rose over that band's own noise, and how much of that still rang 40 to 70 ms later (a snare's wires ring, a kick's click doesn't). Three shapes, kick low, snare in the middle and hats on top, are fitted to every sixteenth at once by non-negative matrix factorization, each free to settle on this kit's own sound. From drums alone, beats 1 and 3 often sound the same. Then the fills and pickups decide, since they lead into beat 1.

    sustainSummary() drumHits() drumDownbeat() drumFold()

    On the page: "Heard as" above the readings, and the drum grid and swing in the drums card. If it heard wrong, pick what it is, and it reads the take again.

    Swing

    Playing the second of each pair of eighths (or sixteenths) late. At 50% it's straight; at 67% it falls on the last third of the beat, a triplet shuffle.

    Non-negative matrix factorization

    Splitting a table of numbers into a few shapes and how much of each is there at each moment. Nothing can go below zero, so the parts can only add up, as sounds do (Lee and Seung, 1999).

12. 20

    Your corrections

    Half, Double, Tap the beat, It's in 3, Bar starts one beat later and Read as (a band, one voice or drums only) don't listen again. They hand the slices it already measured back to finish() with a hint (a tempo, a meter, a shift or a reading), and steps 10 to 19 run again in a moment. Back to its own reading drops the tempo and bar hints. While you listen, the live meters use the same slices: the tempo comes from the last 12 seconds, the chord from pitch classes that fade within about a quarter second, and the key from ones that fade over about 40 seconds.

    finish(hints) liveTempo() liveChord() liveKeyNow()

    Hint

    A fact you supply that the math then takes as given.