Comments (6)
Is there an equivalent API in datafusion?
The equivalent can be specified as part of each external table definition. For example https://datafusion.apache.org/user-guide/cli/datasources.html#s3
CREATE EXTERNAL TABLE test
STORED AS PARQUET
OPTIONS(
'aws.access_key_id' '******',
'aws.secret_access_key' '******',
'aws.region' 'us-east-2'
)
LOCATION 's3://bucket/path/file.parquet';
This isn't quite as good as a secret that can be reused but it should work
from arrow-datafusion.
I am pretty interesting in this idea. I saw duckdb just implements Hugging face Authentications via
CREATE SECRET hf_token (
TYPE HUGGINGFACE,
TOKEN 'your_hf_token'
);
or
CREATE SECRET hf_token (
TYPE HUGGINGFACE,
PROVIDER CREDENTIAL_CHAIN
);
Is there an equivalent API in datafusion?
from arrow-datafusion.
Hey @alamb, need some help while implementing the wildcard functions. Did datafusion support select from a wildcard of external files, e.g., select * from 's3://some-bucket/test/*.parquet'. I read through the doc but failed to find a proper example..
I don't think it supports wildcards but instead the Listing table to read all the files in a directory (that have the correct extension):
https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table
CREATE EXTERNAL TABLE test
STORED AS CSV
LOCATION '/path/to/directory/of/files'
OPTIONS ('has_header' 'true');
from arrow-datafusion.
take
from arrow-datafusion.
PR to reference: https://github.com/duckdb/duckdb/pull/11831/files#diff-e0d4fb8749355dd063169c27338bd119b7814546a06720ee1cd18caf83ad5106
from arrow-datafusion.
Hey @alamb, need some help while implementing the wildcard functions. Did datafusion support select from a wildcard of external files, e.g., select * from 's3://some-bucket/test/*.parquet'. I read through the doc but failed to find a proper example..
from arrow-datafusion.
Related Issues (20)
- Improve lpad udf by using a GenericStringBuilder HOT 1
- DataFusion weekly project plan (Andrew Lamb) - Aug 12, 2024 HOT 1
- Disable `create_default_catalog` when the exist session state has default catalog for `SessionStateBuilder` HOT 1
- Improve performance of `REPEAT` functions HOT 2
- Internal error: Empty iterator passed to ScalarValue::iter_to_array HOT 1
- `optimize_projections` fails when eagerly selecting from a join with ambigious column names HOT 1
- Speed up rpad implementation to use StringBuilder HOT 1
- Remove unused config value `scalar_update_factor` HOT 1
- 在文件prost.rs文件中定义的"pub struct Schema"等结构体,需要使用"#[derive(Serialize)]"属性生成序列化的方法。 HOT 4
- Improve error message for invalid aggregate queries
- Support different aggregate expression (outside SELECT list) in ORDER BY list HOT 2
- Allow suppling a table schema to ParquetExec HOT 7
- Panics in `MIN()/MAX()` aggregate functions (SQLancer) HOT 3
- Internal error in `approx_percentile_cont()` aggregate function (SQLancer) HOT 3
- Invalid aggregate SQL query with `HAVING` can be executed without error (SQLancer-TLP) HOT 3
- Unable to access LogicalPlanBuilder.plan inside a new trail implementation
- Make properties in phyiscal-plan/aggregates public
- COUNT(expr) always returns the COUNT(colname) HOT 7
- Convert built-in `row_number` to user-defined window function
- Improve performance of SUBSTR for StringViewArray HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from arrow-datafusion.