Kaggle lets you pick two submissions for final scoring. Most of the conversation in any competition is about the number on the public leaderboard — how to push it up, what the top teams are doing, whether that 0.001 is real. Almost none of it is about that choice at the end.
At BirdCLEF+ 2026 the choice was the whole thing. I finished 52nd of 4,094 with a private score of 0.94927 — a silver medal, entered solo. On the public board I was 88th. The reshuffle moved me up 36 places while the people around me went down.
That gap is what this post is about.
The scoring asymmetry nobody feels until it’s too late
The public leaderboard is computed on a slice of the test data. The private one — the only one that decides medals — uses the rest, and you never see it until the competition closes.
Everyone knows this. The failure is not ignorance, it’s incentive: you get a number back within minutes of submitting, and you get that number hundreds of times over two months. It is the only feedback with a fast loop. So people optimise it, and by the end a large part of the field has quietly fit their model selection to a few thousand held-out rows.
You find out on the last day whether that was you.
Wall 1: a ceiling that wasn’t about tuning
The task was soundscape classification — identify which bird species are calling in field recordings. Hosted by the Cornell Lab of Ornithology with Google DeepMind, $50,000 prize pool.
I started on the obvious foundation: a pretrained bird-audio embedding model, plus the usual head on top. Six days of tuning, post-processing changes, different inference backends, pseudo-labelling. Everything landed at 0.929 or below. Not “0.929, then 0.931, then 0.934” — a wall, hit from several directions.
The useful conclusion wasn’t a better hyperparameter. It was that I was searching inside a family of models that all made the same mistakes, and no amount of reweighting the same mistakes produces a different answer. Every improvement after this point came from the decision to stop optimising within that family and go find a second signal source that failed differently.
That’s the whole lesson of Wall 1, and it generalises well past Kaggle: when a plateau is flat from every direction you approach it from, the plateau is a property of your hypothesis class, not of your search.
Wall 2: 1,135 teams on the same number
The next ceiling was more interesting because it was social. A well-known public notebook family topped out at 0.950, and by late May there were 1,135 teams sitting on that exact score.
A thousand teams at an identical number is not a coincidence, and it is not a leaderboard — it’s a single model wearing a thousand hats. Everyone had converged onto the same anchor, and their submissions were near-perfectly correlated with each other.
Breaking it needed a leg that was both strong and genuinely decorrelated from the anchor. Strong alone was not enough: I tried several standalone models that scored respectably on their own — 0.947, 0.948, 0.949 — and none of them lifted the blend, because they were making the same errors the anchor already made. Decorrelated alone was not enough either: I trained a model that was beautifully independent (correlation ≈ 0.50 against the anchor) and scored 0.894 by itself, too weak to carry weight.
What finally worked was a light-weight blend: the anchor, plus a public CNN from the community as a sidecar, rank-blended at weight 0.20. That combination reached 0.953 and jumped me from roughly 639th to 72nd in a day.
The decision that actually mattered
Here’s the part I want to argue for, because it’s the part that looks like luck from outside.
With 0.953 in hand, the tempting move is to keep pushing — find 0.954, find 0.955. I spent two days trying, found nothing that beat it, and then stopped pushing entirely and spent the remaining time on the selection problem instead.
The two submissions I locked:
| Slot | What it was | Public | Private |
|---|---|---|---|
| 1 | Anchor + public CNN sidecar, rank-blend at w=0.20 | 0.953 | 0.949 |
| 2 | A different anchor + my own locally-trained model, blend at 0.10 | 0.949 | 0.940 |
Note that slot 2 was worse on the public board than several alternatives I could have picked. That was deliberate.
The pair was not chosen to maximise expected score. It was chosen so that the two submissions fail for different reasons:
- Slot 1 is a bet that the community CNN generalises to the private split.
- Slot 2 contains none of that CNN. It’s the hedge for the world where that bet is wrong — a different anchor, and a leg I trained myself.
Correlation between the two: ≈ 0.938. For comparison, two submissions both built on the popular anchor typically correlate above 0.99. Those pairs are not a hedge; they’re the same bet written down twice, and a thousand teams were holding exactly that.
Since Kaggle scores you on the better of your two, a pair with genuinely different failure modes converts private-board variance from a threat into a free option.
Why 0.20 and not 0.15
A smaller version of the same reasoning, and my favourite detail from the whole competition.
Two candidate weights for the sidecar, w=0.15 and w=0.20, produced the same public score — 0.953 both. Indistinguishable by the feedback everyone was optimising.
I picked 0.20 on two tiebreakers that the public board could not see: it scored higher on a local held-out proxy (AUC 0.99645 vs 0.99614), and it was less correlated with my other slot. Tiny margins, but they were the only evidence available that wasn’t already contaminated by the thing I was trying not to overfit to.
When two options are tied on the metric you can see, break the tie with evidence that metric can’t reach.
What the private board did
The reshuffle came, and my immediate neighbourhood on the final leaderboard looked like this:
rank 50 ▲99
rank 51 ▼35
rank 52 ▲5
rank 53 ▲17
rank 54 ▲34 ← me
rank 55 ▼26
rank 58 ▼43
Almost everyone around me moved, in both directions, by tens of places. That’s what a shakeup looks like from the inside: the ordering was never as precise as the public board’s four decimal places implied.
My two slots landed at 0.949 and 0.940. The hedge, in the end, was not needed — the CNN bet held. But that’s exactly the point, and the reason I don’t read this as luck: had it gone the other way, slot 2 was there. I did not need to be right about which world I was in. I needed both worlds to be survivable.
The gap between my two slots is worth sitting with too: 0.004 apart on the public board, 0.009 apart on the private one. It more than doubled. Four decimal places of public score did not carry four decimal places of information about the split that counted — and that was the number I’d been reading, several times a day, for two months.
(Postscript: the snapshot above is from results day, where I placed 54th, up 34. A later review disqualified two teams above me — final standing 52nd, up 36. That’s the number quoted at the top.)
What I’d take to the next one
- A flat plateau is a statement about your model family, not your search. Change families before you change learning rates.
- A thousand teams on an identical score are one team. If your submission correlates 0.99 with the crowd, you have no independent information, whatever your score says.
- Strong and decorrelated are both necessary. A strong-but-redundant leg adds nothing; an independent-but-weak leg can’t carry weight.
- Your two final slots should be able to fail differently. Two variants of your best model is not a hedge.
- When the visible metric ties, break it with something the metric can’t see. That’s the only evidence left that you haven’t already overfit.
None of this required a bigger model or more compute. It required treating the public leaderboard as what it is: a noisy sample, generous with feedback, and quietly the most effective overfitting trap in the competition.