Comments (9)
Thanks @Rachelint -- I also have some time later this week to help with this issue (if it might help to have one or two examples as we work through #11151)
from arrow-datafusion.
Sounds like it might work -- I think the only way to really know for sure would be to try it out
Thanks, I try it now.
from arrow-datafusion.
Sounds like it might work -- I think the only way to really know for sure would be to try it out
Thanks, I try it now.
@Rachelint if you stop working on this could you please let me know?
Sorry for delay, still working on now, will try to push codes today...
from arrow-datafusion.
take
from arrow-datafusion.
Hi @alamb , I have some questions about the alternatives mentioned in the issue.
It seems the interface should be like:
trait AggregateExpr {
fn output_from_stats(&self, stats: &Statistics) -> Option<ScalarValue> { None }
...
}
As I understand, it is a optimization takes effect when specific aggregate functions(such as max, min, count...) can get the results directly from stats?
from arrow-datafusion.
As I understand, it is a optimization takes effect when specific aggregate functions(such as max, min, count...) can get the results directly from stats?
Yes that is the case
I think output_from_stats
would also be reasonable and potentially more general. One challenge is that the AggregateExpr
might not have access to its argument exprs (and thus it might not know what the argument is) 🤔
from arrow-datafusion.
As I understand, it is a optimization takes effect when specific aggregate functions(such as max, min, count...) can get the results directly from stats?
Yes that is the case
I think
output_from_stats
would also be reasonable and potentially more general. One challenge is that theAggregateExpr
might not have access to its argument exprs (and thus it might not know what the argument is) 🤔
It seems a function expressions
exists, and can help?
and it is used to get min/max from stats now?
datafusion/datafusion/core/src/physical_optimizer/aggregate_statistics.rs
Lines 250 to 257 in 569be9e
from arrow-datafusion.
Sounds like it might work -- I think the only way to really know for sure would be to try it out
from arrow-datafusion.
Sounds like it might work -- I think the only way to really know for sure would be to try it out
Thanks, I try it now.
@Rachelint if you stop working on this could you please let me know?
from arrow-datafusion.
Related Issues (20)
- DataFusion weekly project plan (Andrew Lamb) - Aug 12, 2024
- Disable `create_default_catalog` when the exist session state has default catalog for `SessionStateBuilder` HOT 1
- Improve performance of `REPEAT` functions HOT 2
- Internal error: Empty iterator passed to ScalarValue::iter_to_array HOT 1
- `optimize_projections` fails when eagerly selecting from a join with ambigious column names HOT 1
- Speed up rpad implementation to use StringBuilder HOT 1
- Remove unused config value `scalar_update_factor` HOT 1
- 在文件prost.rs文件中定义的"pub struct Schema"等结构体,需要使用"#[derive(Serialize)]"属性生成序列化的方法。 HOT 2
- Improve error message for invalid aggregate queries
- Support different aggregate expression (outside SELECT list) in ORDER BY list HOT 2
- Allow suppling a table schema to ParquetExec HOT 4
- Panics in `MIN()/MAX()` aggregate functions (SQLancer) HOT 3
- Internal error in `approx_percentile_cont()` aggregate function (SQLancer)
- Invalid aggregate SQL query with `HAVING` can be executed without error (SQLancer-TLP)
- Unable to access LogicalPlanBuilder.plan inside a new trail implementation
- Make properties in phyiscal-plan/aggregates public
- COUNT(expr) always returns the COUNT(colname) HOT 6
- Convert built-in `row_number` to user-defined window function
- Improve performance of SUBSTR for StringViewArray HOT 3
- Empty strings in CSV files aren't being interpreted as null when using a `Dictionary(_, Utf8)`
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from arrow-datafusion.