Data & context
DataTrove
Separates reusable corpus-processing blocks from the infrastructure that executes them.
Overview
Research summary
DataTrove assembles reusable processing blocks into pipelines for large-scale text-data preparation. Readers, filters, deduplication stages and writers can be combined with custom blocks, while filesystem access supports local and remote storage. Execution options include local workers and cluster backends such as Slurm and Ray. Its multi-stage design and resource-conscious processing are intended for corpora that exceed a single machine’s convenient working size. The project also documents synthetic-data workflows and statistics for inspecting the resulting datasets.
Repository summary
- Stars
- 3,366
- Open issues
- Unavailable
- Last push
- 2026-09-30
- Commits, 90 days
- Unavailable
- Repository activity
- Not scored
- Version
- Unavailable
Recorded catalogue figures. View repository data and provenance →
Classification
Pricing & services
Paid services unknown
Whether the provider offers paid products or services has not been established.
Licence scope
Implementation
Recorded implementation details and interfaces for DataTrove.
Implementation details
Python library and supporting tools
- Languages
- Python
- Repository type
- source
Recorded interfaces and capabilities
Licence scope
Repository
Repository snapshots, release information and recorded maintenance signals.
Repository snapshot
- Stars
- 3,366
- Open issues
- Unavailable
- Last push
- 2026-09-30
- Commits, 90 days
- Unavailable
- Repository activity
- Not scored
- Archived
- Not recorded
Repository activity is a snapshot, not a quality or popularity ranking. It combines recent-push freshness (50%), 90-day commits (30%) and issue pressure (20%).
Maintenance and provenance
- Catalogue snapshot
- 2026-10-06
- Stars source
- Recorded fallback
- Last push source
- Recorded fallback
Documentation
Recorded references and research provenance for this entry.
Recorded sources 4
- https://github.com/huggingface/datatrove Project page · Linked repository · Research reference
- https://api.github.com/repos/huggingface/datatrove Research reference
- https://github.com/huggingface/datatrove/blob/main/README.md Research reference
- https://github.com/huggingface/datatrove/blob/main/LICENSE Research reference
Research metadata
- Research date
- 2026-10-02
- Catalogue snapshot
- 2026-10-06
Related tools
Editorially linked entries to explore alongside DataTrove. This is not a ranking.
