Dev Tools
trafilatura
Trafilatura — a Python and CLI tool to crawl the web and extract clean text and metadata for corpora and RAG.
★ 6.9k starsApache-2.0
View guide →Adaptive web scraping framework in Python — from a single request to a full-scale crawl
Scrapling is an open-source dev tools project by D4Vinci (D4Vinci/Scrapling on GitHub). It has 86,348 GitHub stars and 8,855 forks, is released under the BSD-3-Clause license, and was last updated on GitHub on 8 October 2026.
Decide what you want from Scrapling and check the README and requirements to confirm it fits your setup and Python environment.
Clone the repository from GitHub (github.com/D4Vinci/Scrapling) or use the documented install channel if you do not need the source.
Follow the README quickstart to install dependencies and set any required configuration or credentials.
Work through the README's first example end to end, then adapt it to your own project or data.
Selectors adapt when page structure changes, reducing broken scrapers.
Includes a Spiders framework for full-scale crawls, not just single-page fetches.
Advanced fetching with session support and stealth options built on Playwright.
Ships AI-oriented extras, including an MCP server and RAG-building support.
Async session management examples for high-throughput scraping.
Provides a CLI and an interactive shell for quick extraction jobs.
Start by getting the source and entering the project directory.
git clone https://github.com/D4Vinci/Scrapling.git
cd ScraplingInstall the core package, or the fetchers extra for browser-based scraping.
pip install scraplingRequired after installing the fetchers extra, per the README.
scrapling installIt is maintained by D4Vinci and contributors on GitHub, where the repository github.com/D4Vinci/Scrapling is the canonical home for code, issues and releases.
Yes — it is open source under the BSD-3-Clause license. Read the license in the repository for what is permitted, including any conditions on redistribution or commercial use.
No. Scrapling is a local library; optional AI extras use your own model access if you enable them.
Compared with BeautifulSoup or Scrapy: Scrapling adds adaptive selectors and stealth browser fetching on top of a Scrapy-like crawling model.
Clone github.com/D4Vinci/Scrapling, follow the README quickstart to install it, and run the documented first example before adapting it to your own work.
Trafilatura — a Python and CLI tool to crawl the web and extract clean text and metadata for corpora and RAG.
Scrape any website with a prompt: LLM-powered scraping pipelines in a few lines of Python
Skybridge: the full-stack TypeScript framework for building type-safe MCP Apps and ChatGPT Apps with React.
Build machine-learning web demos in Python within minutes — the fastest way to share a model with the world.
A Python metasearch library aggregating many web search services — plus CLI, API and MCP server
Local-first web search, fetch, crawl and research for AI coding agents over MCP — no API keys.
The most-starred other Dev Tools projects in the LibreAI catalogue — the realistic shortlist if Scrapling is not the right fit.
The open source coding agent
The model-definition framework behind modern AI — thousands of pretrained transformers for text, vision, and audio.
Anthropic's agentic coding tool that lives in your terminal and works across your whole codebase
Cross-platform desktop all-in-one manager for Claude Code, Codex, OpenCode and other coding agents
OpenAI's lightweight coding agent that runs locally in your terminal
Free and open — grab the source, star the repo, and start creating.