Code Monkey home page Code Monkey logo

Comments (2)

xuzhangtian avatar xuzhangtian commented on July 1, 2024 1

在 PushProjectIntoTableSourceScanRule.onMatch(RelOptRuleCall call) 方法中源表字段和输入字段数量不一致会调用

        final TableSourceTable newSource =
                sourceTable.copy(
                        newTableSource,
                        newRowType,
                        getExtraDigests(abilitySpecs),
                        abilitySpecs.toArray(new SourceAbilitySpec[0]));

导致 TableSourceTable.getQualifiedName() 方法中会多一个值 “project=[id, birthday, first_name, last_name]”。

    private Set<String> optimizeSourceColumnSet(Set<RelColumnOrigin> inputSet) {
        Set<String> catalogSet = new HashSet<>();
        Set<String> databaseSet = new HashSet<>();
        Set<String> tableSet = new HashSet<>();
        Set<List<String>> qualifiedSet = new LinkedHashSet<>();
        for (RelColumnOrigin rco : inputSet) {
            RelOptTable originTable = rco.getOriginTable();
            List<String> qualifiedName = originTable.getQualifiedName();

            // catalog,database,table,field
            List<String> qualifiedList = new ArrayList<>(qualifiedName);
            catalogSet.add(qualifiedName.get(0));
            databaseSet.add(qualifiedName.get(1));
            tableSet.add(qualifiedName.get(2));

            String field = rco.getTransform() != null ? rco.getTransform() :
                    originTable.getRowType().getFieldNames().get(rco.getOriginColumnOrdinal());
           
            // 这里是否应该改成 qualifiedList.add(3, field)
            qualifiedList.add(field);
            
            qualifiedSet.add(qualifiedList);
        }
        if (catalogSet.size() == 1 && databaseSet.size() == 1 && tableSet.size() == 1) {
            return optimizeName(qualifiedSet, e -> e.get(3));
        } else if (catalogSet.size() == 1 && databaseSet.size() == 1) {
            return optimizeName(qualifiedSet, e -> String.join(DELIMITER, e.subList(2, 4)));
        } else if (catalogSet.size() == 1) {
            return optimizeName(qualifiedSet, e -> String.join(DELIMITER, e.subList(1, 4)));
        } else {
            return optimizeName(qualifiedSet, e -> String.join(DELIMITER, e));
        }
    }

from flink-sql-lineage.

HamaWhiteGG avatar HamaWhiteGG commented on July 1, 2024

@xuzhangtian Very good question, and detailed test cases, many thanks.

This is indeed a bug in parsing transform. Mainly due to the wrong order of parsing transform substitution variables.

I have added your use case to the project source code and passed the unit test.

test case: SimpleTest.java
屏幕快照 2023-04-28 下午4 44 28

You can run it directly or SuiteTest to see all the test results。
屏幕快照 2023-04-28 下午4 47 38

The core code modified by this BUG is to add the RelMdColumnOrigins.buildSourceColumnMap method
屏幕快照 2023-04-28 下午4 51 52

from flink-sql-lineage.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.