Benchmark · Video Globe Challenge 2020
Blind, against the hydrologists.
Eight real videos from France, Norway and Australia, each gauged by 15 to 23 hydrologists with their own tools (Bodart et al. 2023). We developed on videos 1 and 8, froze the agent, the policy and every parameter, then scored videos 2 to 7 once per brain. The scorer reads the reference discharges; the pipeline never does.
Every answer, per video.
Each dot is one hydrologist's discharge, as a signed error against the reference. The shaded band is the reference's own uncertainty. Rank counts the hydrologists plus our answer, by absolute error; 1 is best.
Development videos tuned on, not blind
The refusal suite: zero wrong publications.
Real clips from videos 1 and 8, corrupted in known ways, each with the decision the agent must reach. A wrong PUBLISH on a case that must be refused is the failure that matters. The frozen runs missed three expected decisions, all by refusing too hard; v1.1 fixed them.
Protocol
- Videos 1 and 8 were used for development.
- Station configs for videos 2 to 7 were built from survey inputs only (GRPs, bathymetry, water level and site notes), without opening any results file.
- The agent, the policy and the parameters were frozen:
git tag eval-freeze. - Videos 2 to 7 were scored once per brain. Only
eval/reads the answer key. - Every run, of any version, is appended to
eval/runs.jsonl. - Runs tagged v1.1 came after fixes prompted by the refusal suite. They are not blind, and are reported next to the frozen runs, never in place of them.
- dataset
- Bodart, Le Coz, Jodeau, Hauet (2023), doi:10.57745/KVNIHX, etalab-2.0
- LLM brain
- Claude Sonnet 5.5, tool calling with image input
- results
- eval/results/benchmark.json, served live at /artifacts/public/benchmark.json
- method
- eval/README.md
- not claimed
- That Thalweg beats expert hydrologists in general. Six held-out videos are enough to compare on the same footage, not for a general accuracy claim.