Cloud Embedded Storage layer about latitude HOT 5 OPEN

I'm reading about
READING DB (PostgreSQL/MySQL) and Adding directly to Parquet
https://duckdb.org/2024/01/26/multi-database-support-in-duckdb.html

-- connect to the Postgres instance with the given parameters in read-only mode
ATTACH 'dbname=postgres user=postgres host=127.0.0.1' AS db (TYPE POSTGRES, READ_ONLY);
COPY ⟨table_name⟩ TO 's3://bucket/file.parquet';

The problem with this approach is that DuckDB doesn't have all the connectors we support
Example:
https://www.reddit.com/r/DuckDB/comments/1bp1977/connect_duckdb_with_snowflake_via_adbc/

from latitude.

For Parquet creation/reading this one looks the one that's best maintained
https://github.com/LibertyDSNP/parquetjs

Comparison with others

from latitude.

For DuckDB this one is the official Typescript wrapper around the nodejs client
https://www.npmjs.com/package/duckdb-async

I see duckDB supports writing to parquet directly
https://duckdb.org/docs/data/parquet/overview#writing-to-parquet-files
Not sure if we can do this somehow and avoid using a parquetjs package

from latitude.

Questions:

• Once we've synced to parquet files, how do we tell customers to target this parquet file from their queries?
• How are we gonna show the metadata regarding syncs? Does this mean we are gonna have to do a UI for a product no one is using yet?
• Is this gonna be available in the OSS product? If so, how? We need background jobs which means OSS Latitude becomes a multi-process application that would need probably a docker compose workflow in order to work out of the box in local
• Could we maybe go for an all-frontend approach? where users copy their data into a local duckdb file through some function that we provide, and they can target their queries there

from latitude.

This project could be interesting for connecting to sources that support ODBC connections like Snowflake https://github.com/rupurt/odbc-scanner-duckdb-extension

from latitude.