Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
1.18.0
Description
JsonSerDeTestUtil#testJsonRoundTrip only checks the equality between spec and deserialized object. Some corner cases are detected when serializing the deserialized object again.
static <T> T testJsonRoundTrip(SerdeContext serdeContext, T spec, Class<T> clazz) throws IOException { String actualJson = toJson(serdeContext, spec); T actual = toObject(serdeContext, actualJson, clazz); assertThat(actual).isEqualTo(spec); assertThat(actualJson).isEqualTo(toJson(serdeContext, actual)); // this will eval some corner cases return actual; }
The discovered corner cases are listed as follows.
1. SerDe for AggregateCall
When deserializing the aggregate call, we should check the JsonNodeType to avoid converting null to "null" string.
https://github.com/apache/flink/blob/f9b3e0b7bc0432001b4a197539a0712b16e0b33b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/serde/AggregateCallJsonDeserializer.java#L64
Suggested Fix
JsonNode nameNode = jsonNode.required(FIELD_NAME_NAME); final String name = JsonNodeType.NULL ? null : nameNode.asText();
2. SerDe for RexNode
RexNodeJsonSerdeTest#testSystemFunction should create the temporary system function instead of the temporary catalog function.
https://github.com/apache/flink/blob/f9b3e0b7bc0432001b4a197539a0712b16e0b33b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/serde/RexNodeJsonSerdeTest.java#L209
Suggested Fix
Use functionCatalog#registerTemporarySystemFunction to test.
3. About RexLiteral type
RexNodeJsonSerdeTest#testRexNodeSerde has a test spec as follows
//This will create the literal with DOUBLE as the literal type, and DECIMAL as the broad type of this literal. You can refer to Calcite for more details rexBuilder.makeExactLiteral(BigDecimal.valueOf(Double.MAX_VALUE), FACTORY.createSqlType(SqlTypeName.DOUBLE))
The RexNodeJsonSerializer uses `typeName`(which is DECIMAL) as the literal's type, as a result, the rel data type is serialized as double, but the value is serialized as a string (in case lost the precision)
https://github.com/apache/flink/blob/f9b3e0b7bc0432001b4a197539a0712b16e0b33b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/serde/RexNodeJsonSerializer.java#L197
And then, during the deserialization, according to the JSON, the deserialized literal will assign DOUBLE as the literal type and the broad type of the literal.
This will cause the comparison failure
expected: {"kind": "LITERAL", "value": "1.7976931348623157E+308"} actual: {"kind": "LITERAL", "value": 1.7976931348623157E+308}
Suggested Fix
SARG is a special case and can be coped first, and for the rest type, we can use literal.getType().getSqlTypeName() instead of literal.getTypeName().
// first cope with SARG type if (literal.getTypeName() == SARG) { serializeSargValue( (Sarg<?>) value, literal.getType().getSqlTypeName(), gen, serializerProvider); } else { serializeLiteralValue( value, literal.getType().getSqlTypeName(), gen, serializerProvider); }
Attachments
Issue Links
- Discovered while testing
-
FLINK-31791 FLIP-292: Enhance COMPILED PLAN to support operator-level state TTL configuration
- Closed
- is related to
-
FLINK-25217 FLIP-190: Support Version Upgrades for Table API & SQL Programs
- Open
- links to