Comments (2)
在 PushProjectIntoTableSourceScanRule.onMatch(RelOptRuleCall call) 方法中源表字段和输入字段数量不一致会调用
final TableSourceTable newSource =
sourceTable.copy(
newTableSource,
newRowType,
getExtraDigests(abilitySpecs),
abilitySpecs.toArray(new SourceAbilitySpec[0]));
导致 TableSourceTable.getQualifiedName() 方法中会多一个值 “project=[id, birthday, first_name, last_name]”。
private Set<String> optimizeSourceColumnSet(Set<RelColumnOrigin> inputSet) {
Set<String> catalogSet = new HashSet<>();
Set<String> databaseSet = new HashSet<>();
Set<String> tableSet = new HashSet<>();
Set<List<String>> qualifiedSet = new LinkedHashSet<>();
for (RelColumnOrigin rco : inputSet) {
RelOptTable originTable = rco.getOriginTable();
List<String> qualifiedName = originTable.getQualifiedName();
// catalog,database,table,field
List<String> qualifiedList = new ArrayList<>(qualifiedName);
catalogSet.add(qualifiedName.get(0));
databaseSet.add(qualifiedName.get(1));
tableSet.add(qualifiedName.get(2));
String field = rco.getTransform() != null ? rco.getTransform() :
originTable.getRowType().getFieldNames().get(rco.getOriginColumnOrdinal());
// 这里是否应该改成 qualifiedList.add(3, field)
qualifiedList.add(field);
qualifiedSet.add(qualifiedList);
}
if (catalogSet.size() == 1 && databaseSet.size() == 1 && tableSet.size() == 1) {
return optimizeName(qualifiedSet, e -> e.get(3));
} else if (catalogSet.size() == 1 && databaseSet.size() == 1) {
return optimizeName(qualifiedSet, e -> String.join(DELIMITER, e.subList(2, 4)));
} else if (catalogSet.size() == 1) {
return optimizeName(qualifiedSet, e -> String.join(DELIMITER, e.subList(1, 4)));
} else {
return optimizeName(qualifiedSet, e -> String.join(DELIMITER, e));
}
}
from flink-sql-lineage.
@xuzhangtian Very good question, and detailed test cases, many thanks.
This is indeed a bug in parsing transform. Mainly due to the wrong order of parsing transform substitution variables.
I have added your use case to the project source code and passed the unit test.
test case: SimpleTest.java
You can run it directly or SuiteTest to see all the test results。
The core code modified by this BUG is to add the RelMdColumnOrigins.buildSourceColumnMap method
from flink-sql-lineage.
Related Issues (20)
- 作业级别血缘实现方案探讨
- 血缘解析 使用UUID()时血缘解析 血缘关系错误 flink版本1.14 HOT 1
- failed to deploy with docker-compose HOT 7
- [Feature]Create a universal lineage component for front-end
- mvn clean package出现错误“Failed to run task: 'yarn install' failed” HOT 1
- support flink1.16 HOT 3
- 支持flink java 打包成jar 提交运行血缘
- 当where中包含字段等于常量时,会把源字段优化成常量,导致丢失改字段的血缘关系 HOT 4
- [Discussion] 关于在flink当中添加atlasHook的实现
- PROCTIME()类型的字段血缘关系解析错误 HOT 2
- Field lineage returns an incorrect result HOT 1
- 语法中如果有window tvf会返回空 HOT 1
- [Bug]Flink cep sql column lineage parse error
- UDTF 函数使用别名时解析异常 HOT 5
- 支持带CREATE VIEW 视图流的血缘解析吗? HOT 1
- flinksql lineage result missed one column
- 为什么UT跑的都是报错呢?比如testJdbcCatalog 用例 HOT 1
- Support Flink 1.17 HOT 1
- Push lineage-flink package to the Maven Repository
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from flink-sql-lineage.