Comments (4)
May be relevant: #16624 (comment)
Polars will resolve queries upon collection, not upon DSL creation. This is important as we want to be able to send the query to another machine. At that point the schema will be resolved not when you write the query.
from polars.
Ah should I close this?
from polars.
The call to "schema" will trigger the compute that establishes the execution plan, and that's when it finds out the column isn't present (a call to .explain()
or .collect()
would also raise an error for the same reason). Essentially lazy mode has become even lazier, so that it can better scale across more use-cases (cloud, for instance) 😅
from polars.
I wonder what the implications are for libraries like pandera/patito/etc. They'll have to be built to minimize the number of schema/column calls they make.
I wonder if it's possible to make columns
relatively cheap vs schema
. I believe columns
is used fairly often in a programmatic manner (though perhaps selectors are the ideal/idiomatic way), but I'd imagine that it's a pattern that will be hard to steer users away from.
@ritchie46 @alexander-beedie what are your thoughts on that? I definitely think lots of ETL pipelines involve people doing something along the lines of and in potentially many stages of the pipeline:
exprs = []
for col in ldf.columns:
exprs.append(pl.col(col)...)
from polars.
Related Issues (20)
- Inconsistent Results Between Pandas and Polars using cut (and qcut)? HOT 3
- ExprStringNameSpace replace / replace_all literal flag ignored for dataframes with multiple rows
- `dt.round()` slow/fast path use different rounding HOT 2
- write_parquet with partition_by silently overwrites existing files HOT 7
- rank() on a Series of just 1 null assigns rank=1 to the null value. HOT 2
- read_ndjson ignores provided schema list inner types if values are inferred null HOT 1
- Dependencies not recognized HOT 5
- Inconsistent rolling results when using temporal windows HOT 5
- Polars ignoring rows that are empty in Excel HOT 1
- S3 credentials aren't loaded from `~/.aws/config` if equals aren't padded with spaces
- No non-strict creation of literals HOT 3
- PanicException: index: 8449 out of bounds for len: 1 when using scan csv with schema and include_file_paths
- Fail to compile polars 0.42.0
- Should `str.to_titlecase()` capitalize the letter after an apostrophe? HOT 5
- polars.read_database can not work with duckdb_engine connection. HOT 1
- Build fail HOT 6
- Improve decimal_comma error message
- Add Lateral Column Aliasing support for the SQL interface HOT 2
- maintain_order is a gotcha. Make it true by default? HOT 4
- Creating struct using 'when' HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from polars.