MusicLoRA

Ask a general-purpose LLM for song lyrics and you tend to get the same handful of rhymes and images, common words overused and the expressive vocabulary of real songwriting underused. MusicLoRA is our effort to fix that with a targeted fine-tune instead of more prompting.

Building the training pairs

The interesting part is the data pipeline, which turns finished lyrics into prompt and response pairs:

1. Section splitting. Lyrics are split into their labeled sections, such as verses and choruses.
2. Phrase segmentation. An LLM breaks each section into meaningful phrases of at least two lines and returns them as strict JSON. Robust extraction pulls the JSON out of the reply, and if it's invalid the prompt is reissued at a new random temperature, up to three attempts in all, before that section is skipped.
3. Prompt reconstruction. For each phrase, the model writes five candidate prompts that could have led to it. A second pass, optionally using a different model for a different perspective, picks the best one.

The result is a set of pairs that capture the creative intent behind each phrase, which is what a model needs to learn to write like a lyricist instead of reciting lyrics.

Training

Those pairs fine-tune a pretrained LLM with LoRA, which adjusts only small low-rank weight matrices, keeping training cheap while preserving the base model's knowledge. Model paths are configurable, so base models can be swapped and a separate model can handle candidate selection.