Skip to main content

DuckDB

Availability:
Available Airbyte CloudAvailable Airbyte OSSNot available PyAirbyte
Support Level:
Marketplace
Connector Version:
0.4.20(Last updated 4 days ago)
CDK Version:
0.51.44
Sync Success Rate:
Usage Rate:
caution

Local file-based DBs will not work in Airbyte Cloud or Kubernetes. Please use MotherDuck when running in Airbyte Cloud.

Overview​

DuckDB is an in-process SQL OLAP database management system and this destination is meant to use locally if you have multiple smaller sources such as GitHub repos, some social media and local CSVs or files you want to run analytics workloads on. This destination writes data to the MotherDuck service, or to a file on the local filesystem on the host running Airbyte.

For file-based DBs, data is written to /tmp/airbyte_local by default. To change this location, modify the LOCAL_ROOT environment variable for Airbyte.

Use with MotherDuck​

This DuckDB destination is compatible with MotherDuck.

Specifying a MotherDuck Database​

To specify a MotherDuck-hosted database as your destination, simply provide your database uri with the normal md: database prefix in the destination_path configuration option.

caution

We do not recommend providing your API token in the md: connection string, as this may cause your token to be printed to execution logs. Please use the MotherDuck API Key setting instead.

Authenticating to MotherDuck​

For authentication, you can can provide your MotherDuck Service Credential as the motherduck_api_key configuration option.

Sync Overview​

Output schema​

Each table will contain 3 columns:

  • _airbyte_ab_id: a uuid assigned by Airbyte to each event that is processed.
  • _airbyte_emitted_at: a timestamp representing when the event was pulled from the data source.
  • _airbyte_data: a json blob representing with the event data.

Normalization​

If you set Normalization, source data will be normalized to a tabular form. Let's say you have a source such as GitHub with nested JSONs; the Normalization ensures you end up with tables and columns. Suppose you have a many-to-many relationship between the users and commits. Normalization will create separate tables for it. The end state is the third normal form (3NF).

Features​

FeatureSupported
Full Refresh SyncYes
Incremental - Append SyncYes
Incremental - Append + DedupedNo
NamespacesNo

Performance consideration​

This integration will be constrained by the speed at which your filesystem accepts writes.

Getting Started with Local Database Files​

The destination_path will always start with /local whether it is specified by the user or not. Any directory nesting within local will be mapped onto the local mount.

By default, the LOCAL_ROOT env variable in the .env file is set /tmp/airbyte_local.

The local mount is mounted by Docker onto LOCAL_ROOT. This means the /local is substituted by /tmp/airbyte_local by default.

caution

Please make sure that Docker Desktop has access to /tmp (and /private on a MacOS, as /tmp has a symlink that points to /private. It will not work otherwise). You allow it with "File sharing" in Settings -> Resources -> File sharing -> add the one or two above folder and hit the "Apply & restart" button.

Example:​

  • If destination_path is set to /local/destination.duckdb
  • the local mount is using the /tmp/airbyte_local default
  • then all data will be written to /tmp/airbyte_local/destination.duckdb.

Access Replicated Data Files​

If your Airbyte instance is running on the same computer that you are navigating with, you can open your browser and enter file:///tmp/airbyte_local to look at the replicated data locally. If the first approach fails or if your Airbyte instance is running on a remote server, follow the following steps to access the replicated files:

  1. Access the scheduler container using docker exec -it airbyte-server bash
  2. Navigate to the default local mount using cd /tmp/airbyte_local
  3. Navigate to the replicated file directory you specified when you created the destination, using cd /{destination_path}
  4. Execute duckdb {filename} to access the data in a particular database file.

You can also copy the output file to your host machine, the following command will copy the file to the current working directory you are using:

docker cp airbyte-server:/tmp/airbyte_local/{destination_path} .

Note: If you are running Airbyte on Windows with Docker backed by WSL2, you have to use similar step as above or refer to this link for an alternative approach.

Troubleshooting​

Error message Request failed: (UNAVAILABLE, RPC 'GET_WELCOME_PACK')​

This error may indicate that you are connecting with a 0.10.x DuckDB client (as per DuckDB Destination connector versions >=0.4.0) and your database has not yet been upgraded to a version >=0.10.x. To resolve this, you'll need to manually upgrade your database or revert to a previous version of the DuckDB Destination connector.

Changelog​

Expand to review
VersionDatePull RequestSubject
0.4.182024-08-3144952Update dependencies
0.4.172024-08-2444739Update dependencies
0.4.162024-08-2244530Update test dependencies
0.4.152024-08-1744215Update dependencies
0.4.142024-08-1243755Update dependencies
0.4.132024-08-1043536Update dependencies
0.4.122024-08-0343151Update dependencies
0.4.112024-07-2742753Update dependencies
0.4.102024-07-2042233Update dependencies
0.4.92024-07-1341882Update dependencies
0.4.82024-07-1041521Update dependencies
0.4.72024-07-0941253Update dependencies
0.4.62024-07-0641014Update dependencies
0.4.52024-06-2740215Replaced deprecated AirbyteLogger with logging.Logger
0.4.42024-06-2540354Update dependencies
0.4.32024-06-2340224Update dependencies
0.4.22024-06-2139947Update dependencies
0.4.12024-06-0438959[autopull] Upgrade base image to v1.2.1
0.4.02024-05-30#37515Upgrade DuckDB engine version to v0.10.3.
0.3.62024-05-21#38486[autopull] base image + poetry + up_to_date
0.3.52024-04-23#37515Add resource requirements declaration to metatadat.yml.
0.3.42024-04-16#36715Improve ingestion performance using pyarrow inmem view for writing to DuckDB.
0.3.32024-04-07#36884Fix stale dependency versions in lock file, add CLI for internal testing.
0.3.22024-03-20#32635Instrument custom_user_agent to identify Airbyte-Motherduck connector usage.
0.3.12023-11-18#32635Upgrade DuckDB version to v0.9.2.
0.3.02022-10-23#31744Upgrade DuckDB version to v0.9.1. Required update for all MotherDuck users. Note, this is a BREAKING CHANGE for users who may have other connections using versions of DuckDB prior to 0.9.x. See the 0.9.0 release notes for more information and for upgrade instructions.
0.2.12022-10-20#30600Fix: schema name mapping
0.2.02022-10-19#29428Add support for MotherDuck. Upgrade DuckDB version to `v0.8``.
0.1.02022-10-1417494New DuckDB destination
Was this page helpful?