Is it right?
Scored against reference truth with a numeric tolerance wherever possible — not a vibe check. Negative cases (a window with no event) are first-class, so we catch confident hallucination, not just missed detections.
Why this exists
Geoscience needs proof, not vibes.
AI is arriving in seismology, geodesy and Earth science faster than our ability to check it. A model that writes fluent code can still fetch the wrong station, hallucinate an earthquake in a quiet window, or round a magnitude away. Frontier models are also expensive, and most labs cannot pay a premium for every step of a workflow. We answer three questions at once.
What did it cost?
Every point carries real token cost in dollars. The interesting winner is not the highest score — it is the cheapest system that still clears the quality floor.
Is it reproducible?
Pinned sandboxes, deterministic scorers, versioned skills and a hidden test split mean a score means the same thing next month — and cannot be gamed by training on the answers.
How evals are organised
Three categories of evaluation.
Research is not one kind of task, so the hub is organised by what is being evaluated and how it is scored. Each eval carries a category and tags; the leaderboard below filters on them.
Document-based
Reading and reasoning over the literature: review, critique, translation, interpolation, retrieval-augmented QA, and multimodal figure interpretation. Scored by retrieval metrics, domain-term preservation and citation grounding.
Software-agent
Agents that drive real scientific software: write the detector, run the pipeline, produce the data product. Scored by sandboxed execution and numerical regression against a reference within tolerance.
Research-workflow
Orchestrators that decide which sub-agents to call, in what order, with what dependencies. Scored on the trajectory — the call DAG versus a reference — plus frugal fan-out, not just the final answer.
The evidence
Cost versus performance.
Each model appears twice: without a domain skill (hollow marker) and with it (filled marker), joined by a line. The line is the skill lift. Up is better, left is cheaper — so the best systems sit in the upper-left. Hover any marker for the model version, weights and exact numbers.
Show the numbers as a table
| Model | Eval | Score (no skill) | Score (skill) | Lift | Cost |
|---|
What the live run found: whether you can be frugal depends on the task.
On dv/v parameter choice, free local models (qwen2.5:7b,
llama3.1:8b) reach a perfect 1.00 with the domain skill —
matching claude-haiku at zero cost. On STA/LTA
code generation, every 7B model fails (~0.10): they import the right
ObsPy functions, then hand-roll a buggy detector that crashes. Only the cloud model
writes working code (0.56 → 0.76 with the declustering skill).
Skills rescue a small model on config tasks; they do not rescue one that
cannot write correct numerical code.
Contribute
Build an eval for your own science.
An eval is your agent, your skill, and your golden data — versioned so a score means the same thing a year from now, and split so it cannot be gamed by training on the answers.
1
Keep your agent in your repository
The agent lives with the science code it drives — a specfem agent in
the specfem repo, a noisepy agent in noisepy. EvalHub calls it; it
does not own it. Pin it by commit SHA (never a movable tag) so scores stay
reproducible.
2
Version-control your skill with GitHub tags
A skill is a SKILL.md with frontmatter (name,
version, task_kind). Bump the version whenever the
guidance changes and tag the release — every leaderboard row records the exact
skill version that produced it, so a lift is traceable to the words that caused it.
3
Publish a golden dataset (HF or Zenodo)
Ship the public validation split to Hugging Face or Zenodo with a DOI, in
the strict row schema (id, task_kind, prompt,
gold, scorer_spec, split,
visibility). A sha256 manifest pins the version a score was computed on.
4
Keep a hidden test split
The ranked numbers come from a hidden split whose answers are never published — not in git, not on the hub. Otherwise the benchmark measures memorisation, not capability. Gate it, score it server-side, and rotate it.
Scoring is a declarative spec, not code
Each row carries its own scorer spec, so anyone can score your eval without running your Python — which is what makes a hidden, server-side split possible at all.
{"id": "codameter/param_recommendation/easy-volcano-01",
"task_kind": "numerical_regression",
"split": "validation", "visibility": "public",
"prompt": "Choose the dv/v processing parameters for ...",
"gold": {"estimator": "stretching (TS)", "band": [0.4, 1.0], ...},
"scorer_spec": {"name": "dvv_recovery", "config": {"tolerance": 0.15}}}Full guide: dataset_submission.md — strict schema, the HF/Zenodo pull, and the anti-contamination rules.