Welcome to the Hebrew Handwriting OCR Leaderboard β a community-driven effort to track and compare how well models read Hebrew handwritten manuscripts.
Maintained by ivrit.ai (Hugging Face), a project advancing Hebrew-language AI. Evaluation data is crowdsourced through transcriptor.ivrit.ai, where volunteers transcribe scanned Hebrew handwriting.
Hebrew handwriting is hard for OCR: no vowels, fused prepositions and clitics, cursive hands that vary by writer and period, final-form letters, ligatures, and inconsistent historical spelling. This leaderboard standardizes evaluation, tracks progress, and builds community around Hebrew handwriting recognition.
clean-test10 held-out pages, 10 different hands, 225 lines β each independently transcribed and adjudicated into gold-standard ground truth. Harder, larger batches are coming.
Punctuation and whitespace are canonicalized before scoring (Hebrew text itself is untouched).
The newest vision model from each major provider (Anthropic / OpenAI / Google), across three size tiers. Maintainer-run β no submission flow; results refresh when a provider ships a new model.
Full pipeline β corpus, adjudication tool, eval harness, this page β lives in ocr-eval-ivrit. Want to see line-by-line results? Open the comparison viewer.
Line mode. The model is handed a single cropped line of handwriting and must return its text β the easier task, segmentation already done. Ranked by CER (character error rate, lower is better): the edit distance to the adjudicated line, on normalized text. The number is the median line; the range in parentheses is the 25thβ75th percentile across lines. Outputs with no text are dropped.
| Provider | Model | CER median ββ² |
|---|---|---|
| gemini-flash | 0.119 (0.00β0.50) | |
| gemini-flash-lite | 0.280 (0.11β0.44) | |
| openai | gpt-5.6-sol | 0.440 (0.25β0.57) |
| openai | gpt-5.6-terra | 0.585 (0.44β0.71) |
| anthropic | claude-sonnet-5 | 0.615 (0.45β0.79) |
| anthropic | claude-opus-5 | 0.692 (0.55β0.85) |
| openai | gpt-5.6-luna | 0.735 (0.61β0.84) |
| anthropic | claude-haiku-4-5 | 0.905 (0.80β1.00) |
Full-page mode. The model gets the whole page image at once and returns the entire transcript. Two numbers: Word coverage β is order-independent β the multiset fraction of all gold tokens on the page that appear anywhere in the output, so segmentation and reading-order differences don't count against it. Page CER β is the strict view: each page's gold lines are concatenated in reading order into one reference and edit-distanced against the model's whole output, micro-averaged over pages. A wrong reading order hurts Page CER but not coverage.
| Provider | Model | Word coverage ββΌ | Page CER β |
|---|---|---|---|
| openai | gpt-5.6-sol | 0.462 | 0.400 |
| gemini-flash-lite | 0.453 | 0.353 | |
| anthropic | claude-opus-5 | 0.335 | 0.532 |
| openai | gpt-5.6-terra | 0.270 | 0.599 |
| gemini-flash | 0.215 | 0.764 | |
| anthropic | claude-sonnet-5 | 0.210 | 0.619 |
| openai | gpt-5.6-luna | 0.199 | 1.297 |
| anthropic | claude-haiku-4-5 | 0.197 | 0.793 |
@misc{transcriptor-ivrit,
title = {transcriptor.ivrit.ai: a crowdsourced Hebrew handwriting transcription corpus},
author = {{ivrit.ai}},
year = {2026},
howpublished = {\url{https://transcriptor.ivrit.ai}},
}