ISSUE 092  ·  21 NOVEMBER 2026 ELEN RSS
CodeCity
AUTOMATION № 0090

Scrapling: one scraper, many doors

Scrapling is a Python web-scraping framework that can fetch simple pages, drive a browser when a site resists, and expose the same work from Python, the terminal, and AI tools. In practice it is for turning pages into usable text, links, tables, or cleaned Markdown without forcing you into one mode of use.

Scrapling: ένα εργαλείο για να διαβάζεις τις ιστοσελίδες σαν δεδομένα

Two words first

If you have a website and want to read it as data instead of as a screen, this is Scrapling's territory. It takes a page and turns it into things you can query, filter, and store.

Scrapling is a web-scraping framework. It does more than plain HTML download. It also offers browser-based flows when a site needs JavaScript, and terminal commands when you do not want to write a full program for every small task. The same package covers static pages, dynamic pages, and AI-assisted use. The docs describe it as a tool for “fast HTTP requests, stealth browsers, powerful parsing, and concurrent crawling” docs. Version 0.5.1 is the current stable release shown on PyPI and the repository page PyPI GitHub.

What it does

For a beginner, the win is simple: you write less code to get cleaner output. For an experienced user, the interesting part is that one library gives you several access paths without forcing a single workflow.

In practice, Scrapling does five things well. First, it performs fast HTTP fetches for simple pages docs. Second, it opens a stealthy browser flow when a page needs a real browser or protection such as Cloudflare Turnstile / Interstitials docs. Third, it gives you CSS and XPath selectors to pull only the part you need docs. Fourth, it has terminal extraction for quick file output docs. Fifth, it has an MCP server and a shell for AI use cases and for working from environments like Claude, Cursor, or your own script runner docs.

The most useful detail is that it does not force one way of working. You can use it as a Python library, as CLI from the terminal, or as a server other tools talk to. That flexibility is both its strength and its weight.

What it does not do

It is not a magic button. It does not promise to defeat every anti-bot mechanism. The changelog talks about improved Cloudflare handling and browser sessions, not about universal bypass changelog. So when a site blocks you, you still need your own reasoning, clean testing, and often proxy or browser tuning.

It is also not a tiny package. With extras, it pulls in a fair amount of browser-related dependencies. That means you pay in weight for convenience PyPI. And it does not remove the need to know CSS or XPath. If you do not know what you want to target, the tool will not guess it for you.

Installation

For someone starting simply, the base install is light. For someone who wants browser fetching or CLI support, the right extras matter from the start.

From a project folder or a clean environment:

pip install scrapling

If you want fetchers too, the documentation makes it clear that you need the fetchers extra docs:

pip install "scrapling[fetchers]"

For the terminal shell and interactive use:

pip install "scrapling[shell]"

The docs and PyPI metadata show that the base release requires Python 3.10 or newer PyPI keelbench.

Using it from Python

For practical work, this is the cleanest form. For a beginner, it looks like: get page, extract data, done.

from scrapling.fetchers import Fetcher

page = Fetcher.get('https://quotes.toscrape.com/')
quotes = page.css('.quote .text::text').getall()
print(quotes)

That is the simplest pattern: one request, selectors, result. The documentation shows similar examples for CSS selection, XPath selection, and text extraction docs GitHub.

If you want a browser session instead of a one-off fetch, the package gives you a session object too:

from scrapling.fetchers import StealthySession

with StealthySession(headless=True, solve_cloudflare=True) as session:
    page = session.fetch('https://nopecha.com/demo/cloudflare', google_search=False)
    data = page.css('a').getall()

The value here is that the library does not force you into a different mental model. You stay in the same API, but you switch execution mode depending on how stubborn the page is docs.

CSS, XPath, and targeted extraction

This is where Scrapling becomes useful in everyday work. If you know where the data lives, you do not need to pull the whole page every time.

page.css('#trap h1::text').get()
page.xpath('//span[@class="text"]/text()').getall()

The docs cover CSS, XPath, text search, and regex search docs. That means you can build flows that are more stable than reading entire HTML on every run. For me, that is the most honest part of the tool: it gives you precision, but it asks you to know what you are targeting.

From the terminal

This is the part that makes the library useful to people who do not always want to write Python. Scrapling has a CLI for fast extraction docs.

scrapling extract get https://codecity.gr out.txt
scrapling extract get https://quotes.toscrape.com/ quotes.md --css-selector '.quote'
scrapling extract fetch https://quotes.toscrape.com/js/ quotes.md --wait-selector '.quote' --no-headless

The first is simple download. The second keeps only part of the page. The third shows a browser-based flow for dynamic content docs. The changelog makes it clear that the CLI is a first-class part of the experience, not a side feature changelog.

For pages that need a browser

This is the part that explains why Scrapling exists. A page that loads data after the first HTML does not want only a request. It wants browser state, waits for JavaScript, and sometimes more careful handling docs.

from scrapling.fetchers import StealthyFetcher

page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare', solve_cloudflare=True)

This is not a “push and hope” flow. It is browser fetching with an emphasis on stealth and difficult-page handling docs. In version 0.5.1, the changelog also mentions new remote browser sessions with RemoteSession / AsyncRemoteSession for connecting to an already-open Chromium instance over CDP changelog. That is useful when you do not want to spin up a fresh browser every time.

MCP, shell, and LLM usage

If you look at it from the point of view of an LLM client, Scrapling has a reason to exist beyond the Python API. The docs talk about an MCP server, a shell, and agent-oriented usage docs.

What that means in practice is that you can hand it work from tools that speak MCP and use it as a scraping backend for an assistant. It is not just another library. It is also a tool that can give AI workflows page extraction, browser actions, and saved login state docs.

Claude Code

In Claude Code, the cleanest logic is to use Scrapling as an external tool that the LLM calls through shell or MCP. If you use it as an MCP server, follow the documentation for the ai/mcp-server section docs.

pip install "scrapling[ai]"
scrapling mcp

If your workflow lives in a console, the second step is to connect that server as an available backend in Claude Code. The usage pattern is simple: the assistant asks for a page, Scrapling returns a clean extraction or a browser result docs. That reduces how much custom glue you need to write yourself.

Codex CLI

In Codex CLI, the path is similar but more tool-like. If you want fast extraction, you run the CLI. If you want fuller page access, you attach MCP and let the CLI talk to the server docs.

scrapling extract get https://example.com page.txt
scrapling extract fetch https://example.com page.md --css-selector 'main'

The point is that Codex CLI does not need to know Scrapling internals. It only needs the correct command and the correct output path. That makes the tool easy to drop into pipelines and scripts without extra wrapper code docs.

Cursor

In Cursor, the most natural pattern is to expose Scrapling as a command or MCP tool, depending on how the workspace is set up. The Scrapling documentation explicitly says it works with AI chatbots and agents, and gives Claude/Cursor as examples docs.

scrapling shell

That is useful when you want quick interactive scraping without opening a separate project flow. For more stable Cursor use, the better pattern is to keep the extraction commands simple and let Cursor write the surrounding code. You do not need to turn Scrapling into a framework on top of the editor's framework.

What it costs

We measure three things here: space, model time, and money. Space: the base install (the parsing core only) takes about 14 MB in a clean folder. The moment you add the fetchers extra, the same environment grows to 246 MB, because the browser (playwright, patchright), curl_cffi, and browserforge come along. In other words, the step from reading a simple page to entering a page that uses JavaScript costs roughly 230 MB of disk.

Model memory: because Scrapling is not, first of all, an MCP server, it does not permanently load tool schemas into your context. When you run it from the command line or as a library, the token cost is whatever request you give it — nothing fixed. Only if you stand it up as an MCP server (the ai extra) do you pay the context tax of its tools.

Money: the tool is free and open source (BSD-3-Clause) — no licence and no subscription. The real cost is indirect: the time you spend learning CSS/XPath and, if you hit a page with strong anti-bot protection, the possible need for a proxy — the only expense that can appear. There is no hidden account: you do not hand over a card to run it.

Verdict

Scrapling is a good tool when you want flexibility without changing ecosystems every time a page gets harder. Its strength is that it spans simple fetch, browser automation, and AI-assisted use docs. Its weakness is that it asks a lot from the user: selector knowledge, browser-flow understanding, and acceptance that extras bring weight PyPI.

If you write scraper code often, it will help. If you want a magic button that solves everything without thought, it will not. My judgment: it is more valuable as a serious scraping toolkit than as a simple terminal toy.

Sources:
https://github.com/D4Vinci/Scrapling
https://scrapling.readthedocs.io/en/latest/
https://pypi.org/project/scrapling/
https://scrapling.readthedocs.io/en/latest/changelog.html

Εγκατάσταση

4 ΒΗΜΑΤΑ
  1. pip install scrapling
  2. pip install "scrapling[fetchers]"
  3. pip install "scrapling[shell]"
  4. pip install "scrapling[ai]"
Scrapling: one scraper, many doors · ΑΠΟΤΕΛΕΣΜΑ
ΠΗΓΗ
github.com/D4Vinci/Scrapling