Skip to content

Add DuckDB connector - #153

Open
astr0mfly wants to merge 2 commits into
rbock:mainfrom
astr0mfly:main
Open

astr0mfly wants to merge 2 commits into
rbock:mainfrom
astr0mfly:main

Conversation

@astr0mfly

Copy link
Copy Markdown

DuckDB is an in-process analytical database, so the connector targets
analytical SQL (window functions, CTEs, file readers, Parquet) rather than
trying to mirror sqlite3.

The connector is header-only and built on DuckDB's C API. It provides
connection, connection_config, connection_pool, prepared statements,
transactions, attach and interrupt, plus the insert_or_replace and
insert_or_ignore shorthands, and supports returning, on_conflict and the
using clause of delete_from.

Where DuckDB differs from sqlite3, the connector follows DuckDB:

- parameters are plain ? placeholders bound by position (not ?1, ?2, ...),
- blobs are serialized as '\xNN'::BLOB, since DuckDB has no x'...' literal,
- unsigned_integral maps to the native UBIGINT type instead of being bound
  as int64,
- IS [NOT] DISTINCT FROM uses the SQL standard spelling.

DuckDB parses FOR UPDATE and FOR SHARE but rejects the locking clause with
"SELECT locking clause is not supported", so compatibility_check rejects
for_update at compile time.

DuckDB has no last_insert_rowid(), so last_insert_id() is 0 unless the
statement has a RETURNING clause. Auto-incrementing columns should use a
sequence, as the test tables do.

A database file can only be opened once per process, so the connection
handle keeps one reference counted duckdb_database per path. That is what
makes connection pools work.

DuckDB uses snapshot isolation and does not expose a configurable
per-connection isolation level, so get_default_isolation_level() always
reports serializable and setting anything else throws.

The connector lives in include/sqlpp23/duck_db (namespace sqlpp::duck_db),
while the CMake component is called DuckDB to match the package name and
Sqlpp23DuckDBConfig.cmake. Build with -DBUILD_DUCK_DB_CONNECTOR=ON and make
DuckDB findable, e.g. -DDuckDB_DIR=<dir with duckdb.h and duckdb.lib>.

DuckDB is an in-process analytical database, so the connector targets
analytical SQL (window functions, CTEs, file readers, Parquet) rather than
trying to mirror sqlite3.

The connector is header-only and built on DuckDB's C API. It provides
connection, connection_config, connection_pool, prepared statements,
transactions, attach and interrupt, plus the insert_or_replace and
insert_or_ignore shorthands, and supports returning, on_conflict and the
using clause of delete_from.

Where DuckDB differs from sqlite3, the connector follows DuckDB:

- parameters are plain ? placeholders bound by position (not ?1, ?2, ...),
- blobs are serialized as '\xNN'::BLOB, since DuckDB has no x'...' literal,
- unsigned_integral maps to the native UBIGINT type instead of being bound
  as int64,
- IS [NOT] DISTINCT FROM uses the SQL standard spelling.

DuckDB parses FOR UPDATE and FOR SHARE but rejects the locking clause with
"SELECT locking clause is not supported", so compatibility_check rejects
for_update at compile time.

DuckDB has no last_insert_rowid(), so last_insert_id() is 0 unless the
statement has a RETURNING clause. Auto-incrementing columns should use a
sequence, as the test tables do.

A database file can only be opened once per process, so the connection
handle keeps one reference counted duckdb_database per path. That is what
makes connection pools work.

DuckDB uses snapshot isolation and does not expose a configurable
per-connection isolation level, so get_default_isolation_level() always
reports serializable and setting anything else throws.

The connector lives in include/sqlpp23/duck_db (namespace sqlpp::duck_db),
while the CMake component is called DuckDB to match the package name and
Sqlpp23DuckDBConfig.cmake. Build with -DBUILD_DUCK_DB_CONNECTOR=ON and make
DuckDB findable, e.g. -DDuckDB_DIR=<dir with duckdb.h and duckdb.lib>.
docs/connectors/duck_db.md covers creating connections, sharing a file
between connections and pools, parameter placeholders, the insert_or_*
shorthands, returning, last_insert_id (RETURNING is required), on_conflict,
delete_from ... using, for_update, attach, transactions and isolation
levels, exceptions, multi-statement input, blobs and the data type mapping.

It also lists how the connector differs from the sqlite3 connector, which is
the part users are most likely to trip over.
@rbock

rbock commented Oct 16, 2026

Copy link
Copy Markdown
Owner

Hi, Thanks for sending this PR! I admit, I have never heard of DuckDB before :-)

As of now, I am not super sure whether I would rather maintain the connector in the main library or prefer a separate repository to maintain it.

In any case, here are some comments:

  • I wonder how much interest there would be in a DuckDB connector? According to https://duckdb.org/2024/10/04/duckdb-user-survey-analysis, C/C++ usage of DuckDB seems negligible?
  • If it should be part of this repository, it would have to be integrated into the AppVeyor configuration (or we would need to find an alternative to AppVeyor if that does not work).
  • Currently, all files say Copyright 2026 without stating a copyright holder. Since the vast majority of files are actually 90+% copies of pre-existing files, that does not seem accurate. In most cases, it would be more correct to stick with the copyright notice of the original file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants