Summarize with AI
OpenAI released MultiLangBench, a coding benchmark that asks models to build a SQLite-compatible database engine in 20 programming languages, including COBOL, Fortran, OCaml and Visual Basic. The test suite is on GitHub, and OpenAI also published results from a second evaluation on migrating COBOL code.
OpenAI announced both evaluations on October 19, 2026. It says GPT-6 Astra reaches an average test pass rate of 87.4% on MultiLangBench, up from 26.8% for GPT-5.6 Sol. These "low-resource" languages have far less public code and documentation for models to learn from than Python or JavaScript. They still run critical systems. OpenAI's post cites IBM: COBOL supports more than 40% of online banking systems, 80% of in-person credit card transactions and 95% of ATM transactions. The post also notes that Fortran is used in weather forecasting and nuclear reactor safety analysis.
What it is
- One task for every language. The model must build a SQLite-compatible database engine from scratch in the target language. The engine targets SQLite version 3.40.1.
- Offline. The model gets the SQLite documentation, but no internet access is allowed and all copies of SQLite are removed from its environment.
- Checked by real SQLite. The engine receives a database file and a SQL query and writes a modified database. The real SQLite 3.40.1 then runs a validation query on that file. Tests are adapted from SQLite's own test suite.
- Open. The evaluation setup, test suite and per-language results are on GitHub.
- COBOL migration evaluation. A second evaluation asks models to migrate COBOL programs to C++, Java, Kotlin, Python or Rust. Hidden tests require byte-for-byte parity on observable outputs, and a migration counts only if all tests pass.
MultiLangBench results
At maximum reasoning effort. Average across 20 languages, plus selected languages.
| Language | GPT-5.6 Sol | GPT-6 Luna | GPT-6 Astra |
|---|---|---|---|
| Average (20 languages) | 26.8% | 40.3% | 87.4% |
| COBOL | 10.9% | 17.1% | 72.1% |
| Fortran | 21.4% | 31.4% | 83.5% |
| OCaml | 25.6% | 34.1% | 85.8% |
| Visual Basic | 27.5% | 41.6% | 89.4% |
GPT-6 Astra scores above 83% in 19 of the 20 languages. COBOL is still the hardest. GPT-6 Luna beats GPT-5.6 Sol in every language tested at an estimated average API cost of $0.77 per sample, compared with $3.48 for GPT-5.6 Sol. OpenAI puts that at approximately 78% lower.
COBOL migration results
| Model | Reasoning effort | Accuracy |
|---|---|---|
| GPT-5.6 Sol | Low | 5.4% |
| GPT-6.1 Sol | Low | 86.6% |
| GPT-6.1 Sol | Higher | 97.9% |
At low reasoning effort, GPT-6.1 Sol costs 39% less than GPT-5.6 Sol on this evaluation. OpenAI says the 97.9% result brings the evaluation close to saturation. It adds that using such migrations on production software still needs expert review and testing.
Sources
Did this land? Leave a reaction.
Discussion
0 commentsMembers can join the discussion
Become a member to comment and reply to other members.
Become a MemberNo comments yet
Be the first to start the discussion.