IFBench on Hugging Face: 0 of 23 Rows Are Verified, and Ranks 1 to 6 Sit Inside One 300-Prompt Error Band

2026-11-19来源 dev.toEnglish
IFBench on Hugging Face: 0 of 23 Rows Are Verified, and Ranks 1 to 6 Sit Inside One 300-Prompt Error Band

TL;DR: The official IFBench board on Hugging Face (allenai/IFBench_test) lists 23 rows for 21 distinct models. On Nov 19, 2026 none of the 23 is marked verified, 16 come from pull requests that are still open, and 15 of the 23 values cannot be a whole number of correct prompts out of the benchmark's 300. Ranks 1 to 6 span 3.3 points, which is about 10 prompts, and a rough 95% half-width at n=300 is about 4.6 points. Treat the top of this board as a cluster, not a podium.

IFBench board rows by rank

IFBench is a good benchmark to audit because its design is clear. This post reads the public board JSON the way a reviewer would read a results table: what is each row, where did it come from, and how much of the ordering can 300 prompts support. All numbers below were pulled from the Hugging Face API and from each row's .eval_results file on Nov 19, 2026.

What does IFBench actually measure?

The dataset's own eval.yaml describes it: 300 single-turn prompts from held-out WildChat conversations, paired with 58 out-of-domain constraints (counting, formatting, sentence, word and custom rules, one or two per prompt), each checked by a verification function. The headline metric is prompt-level loose accuracy. A prompt counts as correct only if every one of its constraints passes under loose verification, which also accepts the response with markdown asterisks or its first or last line removed. Scores are percentages, higher is better, and the board declares a single task, prompt_level_loose.

Two consequences follow from that design:

  1. The score granularity is 1/300, so one prompt is 0.333 points.
  2. Every row is supposed to answer the same question on the same 300 prompts. Whether that holds depends on how each row was produced, and the board does not show that.

What does the board JSON look like?

The endpoint is public: https://huggingface.co/api/datasets/allenai/IFBench_test/leaderboard. Each row carries rank, value, verified, a source object, the pullRequest number when the score lives in an unmerged model-repo PR, notes, and num_parameters.

import json, urllib.request def get(url): req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"}) return urllib.request.urlopen(req, timeout=60).read().decode("utf-8") DS = "allenai/IFBench_test" board = json.loads(get(f"https://huggingface.co/api/datasets/{DS}/leaderboard")) print(len(board), "rows,", len({r["modelId"] for r in board}), "distinct models") print("verified:", sum(r["verified"] for r in board)) print("from a pull request:", sum(r.get("pullRequest") is not None for r in board)) whole = sum(abs(r["value"] * 3 - round(r["value"] * 3)) < 0.02 for r in board) print("values that are a whole number of 300 prompts:", whole) print("top 6 spread:", round(board[0]["value"] - board[5]["value"], 2)) 

On Nov 19 this printed 23 rows, 21 distinct models, 0 verified, 16 from a pull request, 8 whole-count values, and a top-6 spread of 3.3. Everything else in this post is derived from those fields plus the raw YAML files they point to.

Is anything on the IFBench board verified?

No. All 23 rows have verified: false. That is normal for a young board and says nothing bad about any model. It does change how to read the table: every number is a claim from the model's own card, a paper, or a lab's own run, not a number the benchmark owner reproduced.

Provenance of the 23 rows:

Source type Rows
Model-card YAML in an open pull request 16
Model-card YAML already on the repo's main branch 3 (ranks 5, 11, 19)
Lab evaluation runs and an EvalEval datastore entry 4 (ranks 20 to 23)

Of the top six rows, five are open pull requests and one (rank 5) is on main. An open PR means the repo owner has not merged the score into the card yet. The board still counts it, which is a reasonable design for coverage, but it means a rank can move if a PR is edited or closed.

Can 300 prompts separate rank 1 from rank 6?

Rank 1 is Qwen/Qwen3.8-2.4T-A95B at 82.8. Rank 6 is Qwen/Qwen3.8-27B at 79.5. The gap is 3.3 points, about 10 prompts.

For a binomial proportion near 0.8 on 300 items the standard error is about 2.3 points, so a naive 95% half-width is about 4.6 points. That is an unpaired approximation. All rows answer the same 300 prompts, and paired differences between two models have a smaller variance than this unpaired figure, so the real uncertainty on a head-to-head gap is smaller. Even so, the test is small, and a 3.3-point spread across six rows is the size of ordinary sampling variation plus ordinary run-to-run variation from sampling temperature.

Ranks 1 to 6 and a rough lower bound

The sizes make the point sharper. The 27B model at rank 6 trails the 2.4T-parameter model at rank 1 by 3.3 points, with roughly one eighty-eighth of the parameters (27.78B against 2,446B in the board's num_parameters field). A board that sorts by score alone cannot show that, and for someone choosing a model to deploy, that ratio matters more than the order of the first six rows. Rank 3, InternScience/Agents-A1 at 80.61 with 35.1B parameters, is also ahead of the 27B Qwen row.

Ties and near-ties add to this:

  • Ranks 8 and 9 both read 76.5, and both carry the date 2026-04-24.
  • Ranks 13 and 14 (two Nemotron checkpoints) differ by 0.38 points, about one prompt.
  • Ranks 15 and 16 (MiniMax-M2.1 and MiniMax-M2.5) both read 70, so their order is a tie-break, not a result.

Are the values counts out of 300?

Not for most rows. If a row came from one pass over the 300 prompts, value * 3 would be an integer. For 15 of 23 rows it is not. Rank 1 is 248.4 prompts, rank 2 is 243.9, rank 3 is 241.83 and rank 4 is 239.4. The most likely explanations are averaging over several runs, a different harness, or a different sample size, and the board has no column to say which. The YAML entries for these rows carry no sample-size field either.

The eight whole-count values are 77, 72, 70, 70, 47, 40, 36.67 and 30. The last four are the Swiss AI rows, whose notes spell out the setup: n=300, repeats=1, temperature, max_tokens, and prompt-level loose accuracy. That is what a row should disclose. The rest of the board mostly does not.

One more row-level detail is worth knowing. Rank 12, MiniMax-M2 at 72, carries the note "Artificial Analysis methodology." It sits above MiniMax-M2.1 and MiniMax-M2.5 (70 each) from the same lab, all three with the date 2026-10-26. A newer checkpoint ranking below an older one is not impossible, but with a different methodology noted on only one of the three, the order of those three rows is not a like-for-like comparison.

What happens when one model has two rows?

The board has 23 rows and 21 distinct models because swiss-ai/Apertus-v1.5-8B and swiss-ai/Apertus-v1.5-70B each appear twice. The note on the 47 and 36.67 rows says thinking is enabled with temperature=0.6, top_p=0.95, max_tokens=8192. The 40 and 30 rows say thinking is disabled with temperature=0.0, max_tokens=2048. Same checkpoints, same 300 prompts, different settings.

Four Apertus rows

The effect of settings is larger than the effect of scale in this slice: the 8B model with thinking on (47) outranks the 70B model with thinking off (30) and also the 70B model with thinking on (36.67). Board order is a function of the checkpoint and of how it was run. That is useful information about instruction following under a token budget, and it is also a reminder that rank positions mix two variables.

How should I read the IFBench board?

A short checklist that works on this board and on others with the same JSON shape:

  1. Check verified first. Zero here, so read every row as a self-reported claim.
  2. Check pullRequest. Sixteen of 23 rows are unmerged, so rows can change.
  3. Compute value * n. If the product is not an integer, ask what was averaged.
  4. Read notes and the YAML. Look for methodology, thinking mode, temperature and token limits.
  5. Convert rank gaps into prompts. At n=300, a 1-point gap is 3 prompts.
  6. Look at num_parameters. Rank gaps with an 88x size difference deserve a different conclusion from rank gaps at equal size.
  7. Group by lab. Qwen holds 6 of the 23 rows and 6 of the top 10, so lab-level effects are part of the picture.

FAQ

What is IFBench? A 300-prompt benchmark of precise, verifiable output constraints, with 58 constraints not seen in training, scored by prompt-level loose accuracy.

Which model is first on the IFBench board on Hugging Face? On Nov 19, 2026 it is Qwen/Qwen3.8-2.4T-A95B at 82.8, from a model-card entry in an open pull request.

Is the IFBench ranking statistically meaningful? The gap between rank 1 and rank 6 is 3.3 points on 300 prompts, inside a rough 4.6-point half-width. The coarse ordering (top cluster, middle cluster, lab runs at the bottom) is solid. The exact order within the top six is not.

Why are some scores not multiples of 1/300? Those values were likely averaged over repeated runs, rounded, or produced with a different harness. The board does not state which.

Does an open pull request count on the board? Yes. Sixteen of the 23 rows come from open PRs, so they appear on the board before the model repo has merged them.

Can I reproduce this? Yes. The snippet above uses only the public API and prints the same counts.

Further reading

Sources: IFBench dataset and eval.yaml, board JSON. Data pulled Nov 19, 2026; boards change daily.

原文:https://www.a8q9.cn/en/article/ifbench-on-hugging-face-0-of-23-rows-are-verified-and-ranks-1-to-6-sit-inside-on-a8a451.html

相关文章