Comments (8)
Do you think it would still be desirable to be able to turn off file-level partioning for CSVs only (and/or to vary it on a per-type basis)?
I'm not familiar enough with Parquet to understand if there would be a functional reason to disable it for that format?
Yes, using the global option is only a quick fix.
I think it's a good idea to add an extra option field for individual files, and later the planner can make better decisions based on per file's newlines_in_values
property (like for now disable parallel execution)
And it can be CSV-specific, parquet seems not relevant here.
from arrow-datafusion.
I haven't dug into the implementation, but I imagine it becomes harder to find the right split point for multi-threaded reading
That is correct, DataFusion parallel CSV scan does:
- For example it gets a 1000B CSV file and wants to read in 2 threads.
- File is split into [0, 500), [500, 1000) two ranges, then probe the actual line boundary and adjust the range
- Let
arrow-rs
handle reading a valid file byte range.
So I think the first step will be to add arrow support
from arrow-datafusion.
I see, so an approach could be to:
-
Addnewlines_in_values: bool
toarrow_csv::reader::Format
. The implementation could use this to consume terminators within quotes.arrow-csv
operates in this way by default. -
Add
newlines_in_values: bool
todatafusion::config::CsvOptions
(anddatafusion::datasource::file_format::csv::CsvFormat
). The implementation could use this in two ways:Pass thenewlines_in_values
flag on tocsv-arrow
.- Disable parallelism in CSV scanning (I don't think it would be possible to reliably determine if you're inside a quoted string without seeing the whole file).~~
from arrow-datafusion.
Turns out arrow-csv
does correctly parse newlines in quoted values, so the issue comes from reading CSVs in parallel. Limiting the target number of partitions to 1 solves the issue.
For example, if I add this to my code before I run ctx.sql("SELECT * FROM <csv including linebreaks>")
then the query executed successfully (without it there are errors about wrong numbers of columns):
ctx.state_weak_ref()
.upgrade()
.unwrap()
.write_arc()
.config_mut()
.options_mut()
.execution
.target_partitions = 1;
From a UX perspective, this setting is quite disconnected from the intent. It also required changing the session-level target_partitions
setting, which (I assume, untested) would adversely affect reads of CSVs that might not be expected to contain linebreaks. A per-CSV newlines_in_values
setting would help to signpost this situation and allow more efficient plans when only some executed CSVs have this requirement.
from arrow-datafusion.
Turns out
arrow-csv
does correctly parse newlines in quoted values, so the issue comes from reading CSVs in parallel. Limiting the target number of partitions to 1 solves the issue.
Thank you for the investigation @connec , if that's the case, setting datafusion.optimizer.repartition_file_scans
to false
might help: https://datafusion.apache.org/user-guide/configs.html
IIRC this option will only limit file scan to one thread, other parts of the plan will still be parallelized using target_partitions
from arrow-datafusion.
Thanks for the pointer! That would certainly be better than disabling parallelization throughout.
Do you think it would still be desirable to be able to turn off file-level partioning for CSVs only (and/or to vary it on a per-type basis)?
I'm not familiar enough with Parquet to understand if there would be a functional reason to disable it for that format?
from arrow-datafusion.
Furthermore, I think there could be value in controlling this on a per-file basis. E.g. if you're working with multiple CSVs, one of which you know contains newlines in values and others that you know do not (and so can benefit from parallelisation).
from arrow-datafusion.
I've opened a PR with the most straightforward implementation of this I could think of. I'd be glad to receive any feedback on that approach.
from arrow-datafusion.
Related Issues (20)
- DataFusion weekly project plan (Andrew Lamb) - Aug 12, 2024
- Disable `create_default_catalog` when the exist session state has default catalog for `SessionStateBuilder` HOT 1
- Improve performance of `REPEAT` functions HOT 2
- Internal error: Empty iterator passed to ScalarValue::iter_to_array HOT 1
- `optimize_projections` fails when eagerly selecting from a join with ambigious column names HOT 1
- Speed up rpad implementation to use StringBuilder HOT 1
- Remove unused config value `scalar_update_factor` HOT 1
- 在文件prost.rs文件中定义的"pub struct Schema"等结构体,需要使用"#[derive(Serialize)]"属性生成序列化的方法。 HOT 2
- Improve error message for invalid aggregate queries
- Support different aggregate expression (outside SELECT list) in ORDER BY list HOT 2
- Allow suppling a table schema to ParquetExec HOT 4
- Panics in `MIN()/MAX()` aggregate functions (SQLancer) HOT 3
- Internal error in `approx_percentile_cont()` aggregate function (SQLancer)
- Invalid aggregate SQL query with `HAVING` can be executed without error (SQLancer-TLP)
- Unable to access LogicalPlanBuilder.plan inside a new trail implementation
- Make properties in phyiscal-plan/aggregates public
- COUNT(expr) always returns the COUNT(colname) HOT 6
- Convert built-in `row_number` to user-defined window function
- Improve performance of SUBSTR for StringViewArray HOT 3
- Empty strings in CSV files aren't being interpreted as null when using a `Dictionary(_, Utf8)`
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from arrow-datafusion.