Инструменты разработчика
Scrapling
Adaptive web scraping framework in Python — from a single request to a full-scale crawl
★ 86,3 тыс. звёздBSD-3-Clause
Смотреть руководство →trafilatura — проект с открытым исходным кодом из категории «Инструменты разработчика» от adbar (adbar/trafilatura на GitHub). У него 6 935 звёзд на GitHub и 452 форка, он распространяется по лицензии Apache-2.0 и последний раз обновлялся на GitHub 6 октября 2026.
Trafilatura — a Python and CLI tool to crawl the web and extract clean text and metadata for corpora and RAG.
Decide your source set (URLs, feeds or crawled sites) and the output format your corpus or RAG pipeline expects.
Clone adbar/trafilatura from GitHub (or use the install path in the run steps below) and skim its README for the parts relevant to your goal.
Install trafilatura from PyPI with pip, as its documentation's installation page describes.
Fetch one article URL and extract it to Markdown, checking that navigation and ads are gone and metadata (title, date, author fields) is present.
Robust, configurable extraction of the key article content and metadata from messy web pages.
Advanced web crawling and text discovery, including feed and sitemap-based gathering for corpus building.
Emit CSV, JSON, HTML, Markdown, TXT or XML to fit corpora, datasets or RAG ingestion.
Processes online and offline input in parallel, so bulk extraction over large URL lists stays practical.
The most efficient open-source library in ScrapingHub's article-extraction benchmark and the best single tool in a published ROUGE-L comparison.
Use it as a Python library, from the command line, or from R, with the same extraction engine.
Get the source from the official GitHub repository for trafilatura.
git clone https://github.com/adbar/trafilatura.git
cd trafilaturaInstall the package from PyPI per the documentation's installation page linked in the README.
pip install trafilatura # see README documentation links for installation detailsUse the core Python functions from the README to fetch a page and extract its main text.
python -c "from trafilatura import fetch_url, extract; print(extract(fetch_url('https://example.com/article'))[:500])" # pattern from READMEadbar, through the adbar/trafilatura repository, with documentation at trafilatura.readthedocs.io.
Yes — the project is open source under the Apache-2.0 license as recorded for this listing. See the license terms for any conditions on redistribution.
No API keys are needed — extraction runs locally in Python; you are responsible for crawling politely and respecting sites' terms and robots rules.
General frameworks fetch pages and leave content cleaning to you. Trafilatura specialises in the extraction step — main text and metadata, benchmarked against other extractors — and pairs well with a separate crawler for discovery.
Clone the repository (or use the documented installer), follow the install and configure steps on this page, and run the smallest real task first — fetch one article URL and extract it to Markdown, checking that navigation and ads are gone and metadata (title, date, author fields) is present.
Adaptive web scraping framework in Python — from a single request to a full-scale crawl
A Python metasearch library aggregating many web search services — plus CLI, API and MCP server
Build machine-learning web demos in Python within minutes — the fastest way to share a model with the world.
Google's Python library for extracting structured data from unstructured text with LLMs, with grounding
A local-first database client with AI SQL help — 40+ databases, desktop, web, Docker and CLI
Unified, efficient fine-tuning of 100+ LLMs and VLMs with a CLI and Web UI — ACL 2024
Другие проекты из категории «Инструменты разработчика» с наибольшим числом звёзд в каталоге LibreAI — реалистичный список альтернатив, если trafilatura вам не подходит.
The open source coding agent
The model-definition framework behind modern AI — thousands of pretrained transformers for text, vision, and audio.
Anthropic's agentic coding tool that lives in your terminal and works across your whole codebase
Cross-platform desktop all-in-one manager for Claude Code, Codex, OpenCode and other coding agents
OpenAI's lightweight coding agent that runs locally in your terminal
Соседние проекты из категории «Инструменты разработчика» в каталоге, ближайшие по числу звёзд к trafilatura — продолжайте изучать полку по обе стороны от него.
Repowise — codebase intelligence: health scores, auto-docs, git analytics and dead-code detection via MCP.
PostgresML — machine learning inside PostgreSQL with GPU acceleration: train, embed and run LLMs in SQL.
Gentle-AI configures the AI coding agents you already use: Claude Code, Cursor, OpenCode, Codex, Pi, and more. Choose persistent memory, Organic-Driven Devel…
A highly extensible Markdown editor featuring version control, AI Copilot, document annotations, mind maps, document encryption, executable code snippets, ch…
AI enabled pair programmer for Claude, GPT, O Series, Grok, Deepseek, Gemini and 300+ models
AI coding assistant for Xcode
Бесплатно и открыто — забирайте исходный код, ставьте звезду репозиторию и начинайте создавать.