ISSUE 086  ·  17 NOVEMBER 2026 ELEN RSS
CodeCity
KNOWLEDGE & MEMORY № 0086

DuckDB: SQL analysis inside your app, without a server

DuckDB is an analytical database that lives inside your application and works without a separate server. It speaks SQL, reads CSV, JSON, and Parquet directly, and gives you fast answers over files without heavy setup.

DuckDB: SQL ανάλυση μέσα στο πρόγραμμα, χωρίς διακομιστή

Two words first

If you have a file full of rows and you want to ask questions about it without building a whole database stack, DuckDB is made for that. It is not a server you call from far away. It is an analysis tool that lives inside your app and works next to the data.

Its core idea is simple. You load the library and run queries where the data already is. You do not start a separate service, you do not wire up networking, and you do not add another thing that has to stay alive. That “inside the program” part is the point. You get SQL without the overhead of a full server setup.

For a beginner, that means you can open a CSV like a table. For an experienced user, it means you can write normal SQL for counting, filtering, grouping, and joins without changing tools. DuckDB is not pretending to be everything for everyone. It is very good at one thing: analytical work on data you already have.

What it does

DuckDB takes large datasets and analyzes them quickly inside a local process. The official site describes it as an analytical SQL database that runs in-process and reads common data formats locally or from cloud storage duckdb.org. The repository says it more bluntly: it does not need a separate server and it offers rich SQL with window functions and complex types GitHub.

For someone who keeps measurements in CSV, the value is immediate. If you have thousands or hundreds of thousands of rows and want totals by day, by customer, or by category, you do not need to move them into another database first. You open them and ask. That matters when your job is not infrastructure management but getting an answer. The person tracking clicks, events, costs, or deliveries wants to reach a result quickly, not go through three import steps.

For someone searching through log files, the value is just as practical. You can load a CSV or JSON log, filter errors, count by hour, and spot patterns without writing a special parser or building an ELT pipeline just to answer one question. DuckDB fits that small, painful need: “I already have the data, give me the answer now.”

For someone who wants to join two files without setting up a database, the gain is even clearer. One file can hold sales and another products. With a join, you get the merged view, calculate averages, find outliers, and move on. You do not need a permanent schema for work that is temporary. That is enough for one-day analysis, quality checks, and tests before you move to something heavier.

DuckDB reads CSV, Parquet, and JSON without requiring you to load them into its own storage first GitHub. That removes steps, time, and duplicate copies. If your data is already in Parquet, the fit is even better, because Parquet is column-oriented. That means the data is stored column by column rather than row by row. For analytical queries, that matters a lot. If you ask only for two columns from a wide table, the engine does not have to read the whole row payload. It reads only what it needs. That reduces I/O, cuts time, and often puts less pressure on memory DuckDB Parquet docs.

Parquet is not just a smaller file. It is a format that matches how analysis works. Instead of bundling every field of a record together, it keeps values organized by column. That helps when you run SUM, AVG, COUNT, filters, and groupings over a few columns from a much larger dataset. That is why DuckDB so often feels faster on Parquet than on CSV. CSV is convenient for exchange, but Parquet is closer to the shape of analysis DuckDB Parquet docs.

DuckDB also gives clean feedback when things go wrong. When a query breaks, it points to the problem and marks the spot. That is not a small thing. In queries with many joins and subqueries, the time saved by a clear error message is real. You do not have to hunt blindly for the mistake.

What it does not do

DuckDB is not a classic database server. It is not built for permanent network traffic, many simultaneous writers, and heavy production use as the central system of an application. Its place is analytical work inside the process that calls it duckdb.org.

It is also not the right answer when your application needs strict server architecture, detailed role management, continuous writes from many clients, and an always-on operational environment. It can connect to other databases and read data from PostgreSQL, MySQL, or SQLite GitHub, but that does not make it the default core of every system. It is an analysis and transformation tool, not a universal database solution.

It does not create clusters, replication, or high-availability architecture on its own. If you want infrastructure with those characteristics, you need another kind of database and another level of operation. DuckDB is the fast tool that sits beside the data and returns answers, not the engine that organizes the whole service ecosystem around it.

It is not a good fit when your main problem is constant writes from many users in real time. In that case, the priority is concurrent write durability and network service behavior, not local analysis speed. There, DuckDB is better as a helper for experiments, quick checks, or intermediate analysis, while production stays on a system designed for that role.

Finally, it does not erase physical limits. If the data grows huge, memory and disk remain real constraints. DuckDB is efficient, but it does not cancel physics. Useful is not the same as magical.

Our hands-on test

In our test, on a file with 50,000 rows, the CSV was 934,185 bytes. Loading and counting took 0.12 seconds. GROUP BY with sum took 0.005 seconds. That shows what “analytical database” means in practice DuckDB docs.

The same dataset in Parquet came out at 152,706 bytes. That is about six times smaller than the CSV. The query on the Parquet file took 0.016 seconds. It is not just a smaller file. It is a format that fits the way DuckDB reads naturally, because both work around columns DuckDB Parquet docs.

In our test, the session used about 62 MB of memory. That matters because it shows the tool can deliver fast analysis without asking for a heavy footprint at this dataset size. Speed is the first gain. Light operation is the second.

The error behavior was clean. A syntax mistake produced a Parser Error: syntax error at or near "WHERE", with a clear marker under the word. Window functions such as rank() OVER worked normally. That shows this is real SQL, not a reduced subset.

Using it from the shell

If you want to try it without a big setup, start simple. The Python library installs with pip and then you open a connection inside your program. You do not need to treat DuckDB like a server. It is a library and a shell in one.

pip install duckdb

In Python, a minimal start looks like this:

python -c "import duckdb; print(duckdb.connect().execute('select 42').fetchall())"

From there, the pattern stays simple. You create a connection, run SQL, and get a result. If you want to read CSV, Parquet, or JSON, you do it inside the same query or with DuckDB’s own tools, without converting the data into something else first DuckDB docs.

For the person who does not yet separate database from file in their head, the best image is this: DuckDB is like putting a very fast analyst on your work table. It does not ask you to move things elsewhere. It looks at them where they already are.

Installation

Installation is one of DuckDB’s quiet strengths. The latest version on PyPI is 2.0.0 and there are ready-made wheels for current Python versions PyPI. The package is described there as an in-process database PyPI.

If you want the line most beginners will remember, it is this:

pip install duckdb

If you want the logic behind that line, it is simple. You do not start a service. You do not open a port. You do not build server config. You take the library and work. The official installation and client documentation lives at duckdb.org/docs DuckDB docs.

Where it helps you most

DuckDB saves you when your data lives in files and you want a fast answer without setting up a whole stack. It is ideal for exploratory analysis, quick tests, local pipelines, notebooks, and situations where setup speed matters as much as query speed DuckDB docs.

It is also useful when your data is already in an analytical format. Parquet fits naturally here because it stores by column and is often smaller and more analysis-friendly than CSV DuckDB Parquet docs. If your data already lives in that shape, DuckDB feels like a tool entering its native environment.

It also helps when you need to bridge two simple files quickly. A CSV with sales and a CSV with customers can be joined right away. A file with events and a file with descriptions can become one result set without building a database for something temporary. That looks small, but in practice it saves a lot of time. Many problems do not need permanent infrastructure. They need the right query.

It is also useful when your work is a chain of checks and confirmations. You can bring data in, clean it, compare it, and produce an output without passing through a separate database server. For a team that does quick audits or for a person checking whether something makes sense before sending it on, that cuts a lot of time.

Where to avoid it

If you are building a live application with many concurrent users writing constantly, this is not your first weapon. If you want a classic production database with network presence, role management, persistent server settings, and heavy operational plumbing, DuckDB is not built to walk in alone and play hero GitHub.

It is also not the right choice when your system must accept a continuous stream of writes from many clients at the same time. In that case, the goal is not local speed but stable service and infrastructure discipline. There, DuckDB is better as a helper analysis tool, not as the central sink for all data.

It does not free you from thinking about size, format, and flow. If data grows quickly, if write rates climb, or if you need long-running service behavior, you need something built for that. DuckDB is a precise tool. It is not the answer to everything.

What stays with me

What stays with me from DuckDB is the feeling that an analytical tool can respect your time. It does not push you to a separate server, it does not force many intermediate steps, and it does not hide its language from you. If you know SQL, you can start quickly. If you do not, you can still see it as a clean way to ask questions of files without moving them somewhere else first.

And the most practical conclusion is this. DuckDB does not want to be the database for everything. It wants to be the shortest route from file to answer. For a lot of everyday work, that is enough, and then some.

What it costs

Tokens: none. DuckDB is not a server with tools declared to a model. There is no schema loaded into every conversation, so there is no context tax. That is a conclusion from the flow, not a token measurement: what you hand it is a command.

On disk: the Python package downloads 14.7 MB and requires Python 3.11 or newer. No separate environment, no server, no second install.

In memory — our own measurement: in our test the session, including the Python process, held about 62 MB while loading 50,000 rows (a 934,185-byte CSV). As the dataset grows, so does the footprint; DuckDB is fast, but it does not cancel memory.

In money: nothing. MIT licence, no key, no account, no subscription. Everything runs on the machine you run it on.

Ours or the vendor's claim: the disk, memory, and timing figures are our own measurements, on a file we built. That it is an in-process analytical database under MIT is the vendor's claim — and we checked it: the licence is indeed MIT and the package is indeed described as in-process.

Sources:

https://github.com/duckdb/duckdb

https://duckdb.org

https://duckdb.org/docs/2.0/data/parquet/overview.html

https://duckdb.org/docs/stable/

https://pypi.org/project/duckdb/

Εγκατάσταση

2 ΒΗΜΑΤΑ
  1. pip install duckdb
  2. python -c "import duckdb; print(duckdb.connect().execute('select 42').fetchall())"
DuckDB: SQL analysis inside your app, without a server · ΑΠΟΤΕΛΕΣΜΑ
ΠΗΓΗ
github.com/duckdb/duckdb