Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-14026

Using unnest on a array with rows having nested rows throws IllegalArgumentException

Details

    • Bug
    • Status: Resolved
    • P2
    • Resolution: Fixed
    • 2.36.0
    • 2.38.0
    • dsl-sql, sdk-java-core
    • None
    • Patch, Important

    Description

      Using unnest with example sql statement like

      "select t.a1, t.a2, t.a3, d.b1, d.b2, d.b4, d.b3.c1, d.b3.c2, d.b3.c3 from test t cross join unnest(t.a4) d"

      on the following dataset throws Illegal argument exception

      Java code snippet


      // define the input row format level3
      Schema level3Type =
      Schema.builder().addInt32Field("c1").addStringField("c2").addDoubleField("c3").build();

      Row level3Row1 = Row.withSchema(level3Type).addValues(1, "row", 1.0).build();
      Row level3Row2 = Row.withSchema(level3Type).addValues(2, "row", 2.0).build();
      Row level3Row3 = Row.withSchema(level3Type).addValues(3, "row", 3.0).build();

      // define the input row format level3
      Schema level2Type =
      Schema.builder().addInt32Field("b1")
      .addStringField("b2")
      .addRowField("b3", level3Type)
      .addDoubleField("b4").build();

      Row level2Row1 = Row.withSchema(level2Type).addValues(1, "row", level3Row1, 1.0).build();
      Row level2Row2 = Row.withSchema(level2Type).addValues(2, "row", level3Row2, 2.0).build();
      Row level2Row3 = Row.withSchema(level2Type).addValues(3, "row", level3Row3, 3.0).build();

      // define the input row format level3
      Schema level1Type =
      Schema.builder().addInt32Field("a1")
      .addStringField("a2")
      .addDoubleField("a3")
      .addArrayField("a4", Schema.FieldType.row(level2Type))
      .build();
      Row level1Row1 = Row.withSchema(level1Type).addValues(1, "row", 1.0,
      Arrays.asList(level2Row1, level2Row2, level2Row3)).build();
      Row level1Row2 = Row.withSchema(level1Type).addValues(2, "row", 2.0,
      Arrays.asList(level2Row1, level2Row2, level2Row3)).build();
      Row level1Row3 = Row.withSchema(level1Type).addValues(3, "row", 3.0,
      Arrays.asList(level2Row1, level2Row2, level2Row3)).build();

      // create a source PCollection with Create.of();
      PCollection<Row> inputTable =
      PBegin.in(p).apply(Create.of(level1Row1, level1Row2, level1Row3).withRowSchema(level1Type));


      Throws following exception

      java.lang.IllegalArgumentException: Row expected 10 fields. initialized with 8 fields

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              abhijeet-lele Abhijeet
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 20m
                  1h 20m