Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-4522

Re-enable CalcRemoveRule

Details

    • Improvement
    • Status: Resolved
    • P2
    • Resolution: Fixed
    • None
    • Not applicable
    • dsl-sql
    • None

    Description

      BEAM-4449 replaces our use of Project and Filter with a compound operation Calc via these rules:

      • FilterToCalcRule.INSTANCE,
      • ProjectToCalcRule.INSTANCE,

      Which then allows these optimizations:

      • FilterCalcMergeRule.INSTANCE,
      • ProjectCalcMergeRule.INSTANCE,
      • CalcRemoveRule.INSTANCE,
      • CalcMergeRule.INSTANCE,

      This always hits assertion errors in the VolcanoPlanner when CalcRemoveRule is invoked, because:

      • LogicalCalc.create always results in a composite collation trait
      • VolcanoPlanner asserts that there are no composite traits
      • There's no simplify() in between the two (I'm not sure there can be)

      The stack trace (Calcite 1.16.0) is this:

      java.lang.AssertionError
      at org.apache.calcite.plan.volcano.VolcanoPlanner.changeTraits(VolcanoPlanner.java:546)
      at org.apache.calcite.plan.RelOptRule.convert(RelOptRule.java:540)
      at org.apache.calcite.rel.rules.CalcRemoveRule.onMatch(CalcRemoveRule.java:65)
      at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:212)
      at org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:650)
      at org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:368)
      at org.apache.calcite.prepare.PlannerImpl.transform(PlannerImpl.java:336)
      at org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner.convertToBeamRel(BeamQueryPlanner.java:135)

      We can actually make progress by disabling CalcRemoveRule in a couple of places. Filing this ticket to track turning it back on since there is no real reason we shouldn't have it.

      Attachments

        Issue Links

          Activity

            People

              amaliujia Rui Wang
              kenn Kenneth Knowles
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 50m
                  50m