Observability & evaluation
BigCode Evaluation Harness
A benchmark harness specialized for generating, executing, and scoring programming-model outputs.
Overview
Research summary
BigCode Evaluation Harness is a Python framework for benchmarking autoregressive code-generation models. It can generate candidate solutions, evaluate previously generated solutions, or perform both stages in one workflow. The project supports Hugging Face model integrations, multi-GPU generation through Accelerate, and benchmark-specific execution and scoring. Docker-based evaluation helps make generated-code execution more reproducible and isolated. It draws inspiration from EleutherAI LM Evaluation Harness but specializes in programming tasks, where judging correctness often requires running code rather than scoring ordinary text.
Repository summary
- Stars
- Unavailable
- Open issues
- Unavailable
- Last push
- Unavailable
- Commits, 90 days
- Unavailable
- Repository activity
- Not scored
- Version
- Unavailable
Recorded catalogue figures. View repository data and provenance →
Classification
Pricing & services
Paid services unknown
Whether the provider offers paid products or services has not been established.
Licence scope
Implementation
Recorded implementation details and interfaces for BigCode Evaluation Harness.
Implementation details
Python benchmark harness with Accelerate integration and Docker evaluation environments
- Languages
- Python
- Repository type
- source
Recorded interfaces and capabilities
Licence scope
Repository
Repository snapshots, release information and recorded maintenance signals.
Repository snapshot
bigcode-project/bigcode-evaluation-harness ↗
- Stars
- Unavailable
- Open issues
- Unavailable
- Last push
- Unavailable
- Commits, 90 days
- Unavailable
- Repository activity
- Not scored
- Archived
- Not recorded
Repository activity is a snapshot, not a quality or popularity ranking. It combines recent-push freshness (50%), 90-day commits (30%) and issue pressure (20%).
Maintenance and provenance
- Catalogue snapshot
- 2026-10-06
Documentation
Recorded references and research provenance for this entry.
Recorded sources 3
- https://github.com/bigcode-project/bigcode-evaluation-harness Project page · Linked repository · Research reference
- https://github.com/bigcode-project/bigcode-evaluation-harness/blob/main/README.md Research reference
- https://github.com/bigcode-project/bigcode-evaluation-harness/blob/main/LICENSE Research reference
Research metadata
- Research date
- 2026-10-02
- Catalogue snapshot
- 2026-10-06