Vocal Syllable Mapper

A closer look at the app, the systems I owned, and the features behind it.

What the app does

Takes a microphone recording or an uploaded track up to about seven minutes and marks every syllable on a scrollable, zoomable waveform.
Groups those syllables into words and estimates line and section breaks, so the output is a rhythmic skeleton of the vocal rather than just a count.
Plays a click track through the Web Audio API on every predicted syllable during playback, so you can hear whether the detection is actually right.
Exports an annotated PNG, or a fill-in-the-blanks text sheet where each word is an underscore blank sized to its syllable count.
Finds real songs with a similar structural fingerprint from a 343-song index, and can draft matching first lines that respect both the syllable and word pattern.

A syllable is defined by its nucleus, almost always a vowel, and sonority theory says exactly one nucleus per syllable. So v2 builds a curve that peaks on vowel-ness, which by construction has one peak per syllable.
V1 instead looked for onset attacks, which fire zero to three times per syllable depending on how many consonants surround it. That is why its per-song error ranged from under one percent to over six hundred.
V1's model was trained on the v1 detector's own output, so the best it could do was reproduce a biased detector. V2 anchors its labels to a syllable count derived from each song's real lyrics instead.
One song was then reviewed by hand at the spectrogram level. That pass found a genuine lyric-to-audio mismatch and rejected noise the automatic labeler would have kept.

Mean F1 of 0.93 on the eight hand-labeled clips, with six of the eight scoring perfectly.
Median F1 of 0.80 and median precision of 0.87 across 100 diverse acapellas.
Word segmentation validated at roughly three percent word-count error using a Viterbi segmenter, not a silence threshold.
Line and section segmentation at 13.9 percent median error, which is why the interface presents it as an estimate rather than a result.

The trained model was ported to a pure NumPy inference path and validated bit for bit against the PyTorch version, so the deployed app never loads torch. That keeps it inside the memory budget of a small host.
Training, tuning, and the offline review tooling still use the full torch and librosa stack, since none of that runs in the request path.
Segmentation parameters were chosen by measuring against ground truth rather than by taking the first value that looked reasonable.

No full song has true hand-labeled ground truth, so every song-length number is measured against the auto-labeler rather than against a person.
The model is conservative: recall of 0.80 trails precision of 0.87.
Line and section boundaries are the weakest feature that ships, and are labeled as approximate in the interface.
Word segmentation weights were tuned on rap acapellas specifically, and singing-heavy or spoken audio has not been separately validated.