Details
-
Bug
-
Status: Resolved
-
P1
-
Resolution: Fixed
-
2.21.0, 2.22.0, 2.23.0, 2.24.0, 2.25.0, 2.26.0, 2.27.0, 2.28.0, 2.29.0, 2.30.0, 2.31.0, 2.32.0, 2.33.0, 2.34.0, 2.35.0
-
None
Description
The join always fail when given a batch column that is Nullable and a streaming column that is Not Nullable. This was discovered in BeamSideInputJoinRel but is likely an issue in the Beam core join library.
Trivial reproduction with a small test diff:
./gradlew :runners:direct-java:needsRunnerTests --tests org.apache.beam.sdk.schemas.transforms.JoinTest.testInnerJoinDifferentKeys
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/JoinTest.java +++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/JoinTest.java @@ -51,7 +51,7 @@ public class JoinTest { .build(); private static final Schema CG_SCHEMA_2 = Schema.builder() - .addStringField("user2") + .addNullableField("user2", Schema.FieldType.STRING) .addInt32Field("count2") .addStringField("country2") .build();
At a lower level:
./gradlew :runners:direct-java:needsRunnerTests --tests org.apache.beam.sdk.schemas.transforms.CoGroupTest.testCoGroupByDifferentFields
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/CoGroupTest.java +++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/CoGroupTest.java @@ -237,7 +237,7 @@ public class CoGroupTest { private static final Schema CG_SCHEMA_2 = Schema.builder() - .addStringField("user2") + .addNullableField("user2", Schema.FieldType.STRING) .addInt32Field("count2") .addStringField("country2") .build();
Attachments
Issue Links
- relates to
-
-
- Resolved
-
- links to
