Leaderboard
Two numbers per system: verdict accuracy (did it call first / later / book-club / insufficient correctly?) and exact accuracy (did it also get the printing number?). We seed it with a transparent baseline and a human-expert reference; every submission is reproducible.
| System | Verdict | Exact | Notes |
|---|---|---|---|
| Human expert (reference) | 100.0% | 100.0% | Ground truth is authored by hand from the canonical rules; a careful specialist scores 100% by construction. |
| Number-line-only baseline | 80.0% | 80.0% | A trivial reader that takes the lowest number in the line and ignores everything else — publisher conventions, book-club tells, reprint wording. Reproducible from the harness below. |
Want on the board? Run the harness below and send your numbers (and per-item outputs) to [email protected].
Get the benchmark
Scoring harness
Ten lines. Plug in your system, get the same two numbers we report.
import json
GOLD = [json.loads(l) for l in open('firstedbench-v1.jsonl')]
def my_system(copyright_page):
# return {'verdict': <first_printing|later_printing|book_club|insufficient>, 'printing': <int|None>}
...
v = e = 0
for item in GOLD:
pred = my_system(item['copyright_page']); gold = item['answer']
vok = pred['verdict'] == gold['verdict']
v += vok; e += vok and pred.get('printing') == gold.get('printing')
print('verdict accuracy: %.1f%%' % (100*v/len(GOLD)))
print('exact accuracy: %.1f%%' % (100*e/len(GOLD)))
Sample items
A few, with the gold verdict and the reason. The full set spans easy line-reading through the traps that separate a careful identifier from a naive one.
FIRST EDITION 10 9 8 7 6 5 4 3 2 1
The lowest number in the number line is 1, so this is a first printing. The stated "First Edition" agrees.
2 4 6 8 10 9 7 5 3 1
Split (odd/even) number lines were common; order is irrelevant. The lowest number, 1, is present, so it is a first printing.
FIRST EDITION 5 6 7 8 9 10 11
"First Edition" describes the edition, which is retained on later printings. The number line's lowest value is 5, so this is the 5th printing of the first edition — not the sought-after first printing.
Random House, New York FIRST EDITION 9 8 7 6 5 4 3 2
Random House first printings state "First Edition" and carry a number line whose lowest number is 2 (the 1 is not used). With the "First Edition" statement present, this is a first printing.
What’s in it
| Category | Items |
|---|---|
| number line basic | 8 |
| edition vs printing | 3 |
| number line direction | 2 |
| number line partial | 2 |
| number line alternating | 2 |
| publisher convention rh | 2 |
| book club | 2 |
| reprint wording | 2 |
| insufficient | 2 |
Ground truth follows the canonical rules explained in our number-line guide and demonstrated by the First-Edition Checker. It is identification only — never appraisal.
Questions
How is the ground truth decided?
By hand, from the same canonical rules our public tools use: the lowest number in the number line is the printing; a line reaching 1 is a first printing; Random House first editions carry "First Edition" with a line ending in 2; book-club editions are identified by their physical tells; and an "edition" statement does not by itself prove a first printing. Every item ships with its rationale.
What do the scores mean?
Verdict accuracy is the share of items whose predicted verdict matches. Exact accuracy also requires the printing number to match. The number-line-only baseline shows how far a naive reader gets before the traps — book-club look-alikes, the Random House convention, edition-vs-printing — start to bite.
How do I submit a result?
Run the harness on the JSONL, then send your system name, the two accuracy numbers, and (ideally) your per-item outputs to [email protected] or open a pull request. We publish reproducible results and note the method.
Is this an appraisal benchmark?
No. FirstEdBench measures identification of edition and printing only. It never involves value or price. The New Mexico Literacy Project is a for-profit reuse business in the Albuquerque metro; it does not appraise books.
Have the books themselves, not just the puzzle?
We pick up books and media across the Albuquerque metro — free — and keep them out of the landfill.
Schedule a free pickup