Dev Tools
Scrapling
Adaptive web scraping framework in Python — from a single request to a full-scale crawl
★ 86.3k starsBSD-3-Clause
View guide →Trafilatura — a Python and CLI tool to crawl the web and extract clean text and metadata for corpora and RAG.
trafilatura is an open-source dev tools project by adbar (adbar/trafilatura on GitHub). It has 6,935 GitHub stars and 452 forks, is released under the Apache-2.0 license, and was last updated on GitHub on 6 October 2026.
Decide your source set (URLs, feeds or crawled sites) and the output format your corpus or RAG pipeline expects.
Clone adbar/trafilatura from GitHub (or use the install path in the run steps below) and skim its README for the parts relevant to your goal.
Install trafilatura from PyPI with pip, as its documentation's installation page describes.
Fetch one article URL and extract it to Markdown, checking that navigation and ads are gone and metadata (title, date, author fields) is present.
Robust, configurable extraction of the key article content and metadata from messy web pages.
Advanced web crawling and text discovery, including feed and sitemap-based gathering for corpus building.
Emit CSV, JSON, HTML, Markdown, TXT or XML to fit corpora, datasets or RAG ingestion.
Processes online and offline input in parallel, so bulk extraction over large URL lists stays practical.
The most efficient open-source library in ScrapingHub's article-extraction benchmark and the best single tool in a published ROUGE-L comparison.
Use it as a Python library, from the command line, or from R, with the same extraction engine.
Get the source from the official GitHub repository for trafilatura.
git clone https://github.com/adbar/trafilatura.git
cd trafilaturaInstall the package from PyPI per the documentation's installation page linked in the README.
pip install trafilatura # see README documentation links for installation detailsUse the core Python functions from the README to fetch a page and extract its main text.
python -c "from trafilatura import fetch_url, extract; print(extract(fetch_url('https://example.com/article'))[:500])" # pattern from READMEadbar, through the adbar/trafilatura repository, with documentation at trafilatura.readthedocs.io.
Yes — the project is open source under the Apache-2.0 license as recorded for this listing. See the license terms for any conditions on redistribution.
No API keys are needed — extraction runs locally in Python; you are responsible for crawling politely and respecting sites' terms and robots rules.
General frameworks fetch pages and leave content cleaning to you. Trafilatura specialises in the extraction step — main text and metadata, benchmarked against other extractors — and pairs well with a separate crawler for discovery.
Clone the repository (or use the documented installer), follow the install and configure steps on this page, and run the smallest real task first — fetch one article URL and extract it to Markdown, checking that navigation and ads are gone and metadata (title, date, author fields) is present.
Adaptive web scraping framework in Python — from a single request to a full-scale crawl
A Python metasearch library aggregating many web search services — plus CLI, API and MCP server
Build machine-learning web demos in Python within minutes — the fastest way to share a model with the world.
Google's Python library for extracting structured data from unstructured text with LLMs, with grounding
A local-first database client with AI SQL help — 40+ databases, desktop, web, Docker and CLI
Unified, efficient fine-tuning of 100+ LLMs and VLMs with a CLI and Web UI — ACL 2024
The most-starred other Dev Tools projects in the LibreAI catalogue — the realistic shortlist if trafilatura is not the right fit.
The open source coding agent
The model-definition framework behind modern AI — thousands of pretrained transformers for text, vision, and audio.
Anthropic's agentic coding tool that lives in your terminal and works across your whole codebase
Cross-platform desktop all-in-one manager for Claude Code, Codex, OpenCode and other coding agents
OpenAI's lightweight coding agent that runs locally in your terminal
Free and open — grab the source, star the repo, and start creating.