StanLyric

Technical implementation

StanLyric is a lyric-first music search project. Its first goal is simple: type a lyric fragment and find the songs most likely to contain it. It runs BM25 in the browser from a static search artifact, so the page does not need a backend server or live API. StanLyric breaks the fragment into tokens, scores the lyric corpus with BM25-Okapi, and explains why the top candidate was retrieved. Find the code for training the song name retrieval model here.

The embedding atlas adds a second retrieval view over the cleaned 36,545-song corpus. Cohere Embed v4 maps each song to 1,024 dimensions, and 3D UMAP provides the browser coordinates. Projection quality is reported using trustworthiness and original-space neighbor overlap; the first three PCA components serve as a transparent linear explained-variance baseline. The page ships quantized coordinates, song metadata, six strong graph neighbors per song, and compact hierarchy assignments rather than dense vectors.

The semantic structure is a strict three-level Leiden hierarchy built with the Constant Potts Model. A multiseed resolution sweep selects 139 middle Communities using adjusted Rand agreement, normalized variation of information, adjacent-resolution stability, and balance constraints. Those Communities are aggregated into 20 broad Regions. Large Communities are then split only when the proposed Neighborhoods pass minimum-size, seed-stability, internal-edge-retention, embedding-cohesion-gain, and topic-separation gates. This produces 175 Neighborhoods, with every Neighborhood contained by exactly one Community and every Community contained by exactly one Region.

Each hierarchy level receives the same interpretation contract. Binary song-incidence c-TF-IDF finds distinctive unigrams and bigrams without letting repeated choruses multiply their weight. Corpus prevalence lift and smoothed log-odds separate characteristic language from merely frequent lyric vocabulary. Representative songs are nearest to each node’s centroid in the original 1,024-dimensional cosine space, while boundary songs place substantial graph strength outside the node. Cohesion, weighted conductance, internal edge strength, artist diversity, sampled cosine silhouette, per-song assignment stability, and weighted local-neighbor agreement remain visible as diagnostics.

The readable hierarchy labels are generated in a separate offline OpenAI pass. For each of the 334 hierarchy nodes, StanLyric packages the node’s top terms, NMF topic bridges, representative and boundary songs, artist counts, graph metrics, and parent/child/sibling context, then asks GPT-5.5 for a title, one-line summary, short description, confidence flag, and evidence list. These LLM outputs are presentation metadata layered on top of the deterministic graph pipeline; they do not change song membership, stability, or any quantitative diagnostic.

StanLyric is an information retrieval system for a lyrics corpus. Each song is one document, and the user-provided lyric fragment is one query. The current version uses BM25-Okapi, short for Best Matching 25, because it is lightweight, interpretable, and especially strong when the query contains rare phrase fragments or distinctive words.

For a query \(Q\) and song document \(D\), the implementation adds one contribution for each query term \(q\):

Here, \(f(q,D)\) is the raw term frequency: how often the query word appears in that song’s lyrics. The explanation calls the fraction multiplying IDF the BM25 TF weight. It is the saturated, document-length-normalized version of raw TF, so it does not need to lie between zero and one. The interface also exposes query TF, the number of times the term appears in the submitted fragment. The exact contribution is therefore shown as:

Query TF is one for a term that appears once in the query, so it usually leaves the simpler BM25 TF weight times IDF calculation unchanged.

The inverse document frequency is based on how many of the \(N\) songs contain the term:

The value \(n(q)\) is the number of songs containing \(q\). This gives more weight to unusual words that occur in relatively few songs; a word such as rabbit is therefore more useful than a common word such as the. During offline export, exceptionally common terms whose raw IDF would be negative are assigned a small positive floor based on \(\epsilon=0.25\) and the corpus-average IDF. Repeating a term helps, but the term-frequency fraction in the BM25 formula saturates its contribution, so ten occurrences are not treated as ten times stronger than one.

The denominator also normalizes for document length. Without it, long lyrics would tend to score highly simply because they contain more words and have more chances to match. The ratio \(\lvert D\rvert/\operatorname{avgdl}\) compares a song’s token count with the corpus average. In the current deduplicated 36,545-song artifact, the average document length is approximately 266 tokens.

StanLyric uses \(k_1=1.5\) and \(b=0.75\). The \(k_1\) parameter controls how quickly repeated term frequency reaches diminishing returns. The \(b\) parameter controls the strength of length normalization: \(b=0\) would ignore document length, while \(b=1\) would apply the full normalization. A value of 0.75 provides substantial normalization without letting length dominate the score.

The offline pipeline builds a browser-ready artifact from the prepared StanLyric corpus. The artifact stores song metadata, document lengths, inverse document frequency values, and an inverted index of token frequencies. At runtime, the browser tokenizes the query and computes BM25 scores only for matching postings. This keeps the portfolio page static while still allowing interactive retrieval.

The explanation panel shows which query terms appeared in the retrieved song, which were missing, and the exact scoring path for every matched term: raw TF, BM25 TF weight, query TF, IDF, and their resulting contribution. All these efforts are aimed at making the retrieval system more transparent!

The lyrics corpus comes from the Lyrics-MIDI-Dataset on Hugging Face.