- test macro-F1, best LoRA ensemble
- 0.9513
- fine-tuning regimes compared
- 5
- rounds of evidence-driven iteration
- 4
The question
Financial PhraseBank's all-agree subset is small and clean, so most models land in the same narrow band. The interesting question is not the leaderboard number but what explains the last few points: training regime, tokenizer, ensembling, or the model's general-domain pretraining.
Approach
- Five regimes. Pretrained baseline, frozen backbone with a trained head, a custom head, full fine-tuning, and LoRA, with trainable-parameter counts tracked for each.
- Four rounds. Minimum viable runs, then changes motivated by validation diagnostics, then statistical checks with paired bootstrap, then multi-seed stability, calibration, ensembles, a vocabulary ablation, and per-sample error forensics.
Results
- LoRA comes close to full fine-tuning with a fraction of the trainable parameters; the best single LoRA run reaches about 0.95 test macro-F1 and the best LoRA ensemble 0.9513.
- Temperature scaling lowers calibration error without changing accuracy.
- Larger and mixed ensembles do not move past the best homogeneous ensemble.
- Adding finance vocabulary to the tokenizer changes results only within seed noise.
What I learned
With one 70/15/15 split of a small dataset, the bootstrap interval is wide enough that round-to-round gains should not be over-claimed. The errors every model gets wrong seem to need knowledge of financial discourse, which points to a pretraining-domain gap rather than anything fine-tuning can fix.