Implement `hf://` / "hugging face" integration in datafusion-cli about arrow-datafusion HOT 6 OPEN

Is there an equivalent API in datafusion?

The equivalent can be specified as part of each external table definition. For example https://datafusion.apache.org/user-guide/cli/datasources.html#s3

CREATE EXTERNAL TABLE test
STORED AS PARQUET
OPTIONS(
    'aws.access_key_id' '******',
    'aws.secret_access_key' '******',
    'aws.region' 'us-east-2'
)
LOCATION 's3://bucket/path/file.parquet';

This isn't quite as good as a secret that can be reused but it should work

from arrow-datafusion.

I am pretty interesting in this idea. I saw duckdb just implements Hugging face Authentications via

 CREATE SECRET hf_token (
    TYPE HUGGINGFACE,
    TOKEN 'your_hf_token'
 );

or

 CREATE SECRET hf_token (
    TYPE HUGGINGFACE,
    PROVIDER CREDENTIAL_CHAIN
 );

Is there an equivalent API in datafusion?

from arrow-datafusion.

Hey @alamb, need some help while implementing the wildcard functions. Did datafusion support select from a wildcard of external files, e.g., select * from 's3://some-bucket/test/*.parquet'. I read through the doc but failed to find a proper example..

I don't think it supports wildcards but instead the Listing table to read all the files in a directory (that have the correct extension):

https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table

CREATE EXTERNAL TABLE test
STORED AS CSV
LOCATION '/path/to/directory/of/files'
OPTIONS ('has_header' 'true');

from arrow-datafusion.

take

from arrow-datafusion.

PR to reference: https://github.com/duckdb/duckdb/pull/11831/files#diff-e0d4fb8749355dd063169c27338bd119b7814546a06720ee1cd18caf83ad5106

from arrow-datafusion.

Hey @alamb, need some help while implementing the wildcard functions. Did datafusion support select from a wildcard of external files, e.g., select * from 's3://some-bucket/test/*.parquet'. I read through the doc but failed to find a proper example..

from arrow-datafusion.