Most competitions give you a metric and let you chase it. NeuroGolf gave you a metric that punished you for existing.
The task was ARC — abstract reasoning puzzles, where you get a handful of input grid → output grid examples and have to infer the transformation. Colours propagate along a pipe, shapes complete their own symmetry, blocks fall to the bottom. Four hundred of them.
The twist: you may not submit code. For each task you ship a neural network as an ONNX file, the server runs it, and it has to get every example right.
The second twist: solving it is only the entry fee. Score per task is
25 − ln(cost) cost = parameter count + bytes of intermediate computation
So once your network is correct, the entire competition is about making it smaller.
I finished 208th of 2,963, solo — a bronze medal. This post is about the single practice that mattered most, which had nothing to do with building networks.
What a logarithm does to your incentives
Look at that scoring function for a second, because it sets the whole texture of the competition.
ln is brutally flat. Halving your cost — a real engineering win, the kind you’d be pleased with anywhere else — earns you ln(2) ≈ 0.69 points. To gain a single-digit improvement of, say, 2.3 points on one task, you need to shrink that task’s cost by a factor of ten.
Across four hundred tasks, a total in the 7,000s works out to roughly 18.5 points per task, which back-solves to an average cost around 665. Not 665 parameters — 665 for parameters and every intermediate byte the graph touches while running.
At that scale you are not tuning a model. You are counting bytes, and the difference between rank 200 and rank 100 is a long tail of individually unglamorous savings.
Which means the expensive mistake in this competition is not a bad idea. It’s a misunderstanding of the grader that silently taxes every one of your four hundred files at once.
I made that mistake twice before I learned to stop making it.
Lesson one: a name is not a semantics
Early on, an audit of a public submission package reported something alarming: 319 of the tasks appeared to be bare Identity operators. Identity. As in, return the input unchanged.
If true, that would mean a large fraction of the benchmark was free — that the “puzzles” mostly asked you to hand back what you were given.
It wasn’t true. A direct check of the raw grids found the number of tasks where input == output was zero.
The resolution is a nice piece of ONNX trivia. The operator was golf::Identity — not the standard ONNX Identity, but a custom FunctionProto in the competition’s own domain, with a full computation graph packed inside it. The name told you nothing. The body was doing all the work.
A name is not a semantics. Read the bytes, not the label.
That reads as obvious written down. It was not obvious at 2am when a plausible-sounding audit claimed a third of the competition was a freebie.
Lesson two: the grader is not a constant
Then the real one.
I submitted 4890.34 and went to sleep. I woke up to 1119.12. A drop of 3,771 points, with no action from me — Kaggle had swapped in a new scorer overnight, and 317 of my 394 files were now scoring zero.
Root cause, once the files were opened at the byte level: my build inlined custom function bodies into the graph, but left behind an orphaned ('golf', 1) entry in the model’s opset_import list. The old scorer tolerated a declared-but-unused custom opset. The new one rejected it outright.
The functions were inlined. The declaration saying “this model uses the golf domain” was still sitting there, referring to nothing. One stale line of metadata, 317 files, 3,771 points.
Deleting it restored the score to 4954 — above where I’d started — by the next day.
The lasting effect wasn’t the fix. It was the reframe: the scorer is part of the system under test, and it can change while you sleep. After that, a mid-competition change to how cost was computed landed as a routine adaptation rather than a crisis.
The practice: calibrate, then predict
Both incidents came from the same root — I was reading the leaderboard as a verdict on my work, when it was really a measurement from an instrument I had never calibrated.
So I calibrated it. I could score my files locally, and local scores tracked server scores closely but not identically. Eight data points were enough to pin the relationship:
server score ≈ local score × 0.9135 residual error: 0.017%
One constant. That is all it took to convert the leaderboard from an oracle into an instrument.
From then on, every submission carried a pre-registered prediction: before pushing, write down the score you expect. Then:
- Prediction lands → your model of the system is correct. Proceed.
- Prediction misses → there is something about the system you do not yet know, and you have just been told exactly where to look.
The second outcome is the valuable one. A miss is a free, precisely-located bug report about your own understanding. Without a written prediction, that same submission is just a number that felt lower than you hoped, and you have nothing to investigate — you’ll rationalise it and move on.
What calibration buys you: a worked example
Late in the competition I was tracking an intermittent fault. Certain submissions would randomly lose about 17.61 points — a task that was apparently non-deterministic on the server’s hidden test. I never conclusively identified which one. What I did have was its exact magnitude.
On the final night, a submission predicted at 7428.53 came back at 7410.97. Under normal circumstances that’s the moment you panic: did my last change break something? Do I roll back with hours left?
Instead the gap decomposed, in about thirty seconds:
7428.5265 predicted
− 17.6136 the known intermittent fault
+ 0.0571 the new increment, working as intended
──────────────
7410.97 observed ✓
Closed to two decimal places. My change had worked exactly as predicted; a known die had rolled badly on top of it. No rollback, no lost hours, no decision made out of fear.
That is the whole return on calibration. Not that it prevents bad outcomes — it didn’t prevent this one. It’s that an unexplained drop and an explained drop demand completely different responses, and without a prediction you cannot tell them apart.
The transferable part
The competition-specific details expire the moment the competition ends. These don’t:
- A pre-registered prediction turns every action into an experiment. Same cost, strictly more information — and the misses teach you more than the hits.
- Calibrate the instrument before trusting the reading. Eight data points bought a constant that made every subsequent number interpretable.
- A name is not a semantics. When a label and a behaviour disagree, the bytes are right.
- Anything you did not build can change without telling you — graders, APIs, upstream data. Treat the evaluation harness as part of the system under test.
- Know the shape of your reward function. Under a logarithm, a 2× win is nearly worthless and only order-of-magnitude changes register. That should decide what you work on, before you work on it.
The last one is worth generalising past competitions. Plenty of engineering effort goes into wins that are real, measurable, and — given the actual shape of the objective — worth almost nothing. It pays to check which regime you’re in before you spend the week.