Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-31917

Loss of Idempotence in JsonSerDe Round Trip for AggregateCall and RexNode

    XMLWordPrintableJSON

Details

    Description

      JsonSerDeTestUtil#testJsonRoundTrip only checks the equality between spec and deserialized object. Some corner cases are detected when serializing the deserialized object again.

      static <T> T testJsonRoundTrip(SerdeContext serdeContext, T spec, Class<T> clazz)
                  throws IOException {
              String actualJson = toJson(serdeContext, spec);
              T actual = toObject(serdeContext, actualJson, clazz);
      
              assertThat(actual).isEqualTo(spec);
              assertThat(actualJson).isEqualTo(toJson(serdeContext, actual)); // this will eval some corner cases
              return actual;
          }
      

      The discovered corner cases are listed as follows.

      1. SerDe for AggregateCall

      When deserializing the aggregate call, we should check the JsonNodeType to avoid converting null to "null" string.
      https://github.com/apache/flink/blob/f9b3e0b7bc0432001b4a197539a0712b16e0b33b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/serde/AggregateCallJsonDeserializer.java#L64

      Suggested Fix
      JsonNode nameNode = jsonNode.required(FIELD_NAME_NAME);
      final String name = JsonNodeType.NULL ? null : nameNode.asText();
      
      2. SerDe for RexNode

      RexNodeJsonSerdeTest#testSystemFunction should create the temporary system function instead of the temporary catalog function.
      https://github.com/apache/flink/blob/f9b3e0b7bc0432001b4a197539a0712b16e0b33b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/serde/RexNodeJsonSerdeTest.java#L209

      Suggested Fix

      Use functionCatalog#registerTemporarySystemFunction to test.

      3. About RexLiteral type

      RexNodeJsonSerdeTest#testRexNodeSerde has a test spec as follows

      //This will create the literal with DOUBLE as the literal type, and DECIMAL as the broad type of this literal. You can refer to Calcite for more details
      rexBuilder.makeExactLiteral(BigDecimal.valueOf(Double.MAX_VALUE), FACTORY.createSqlType(SqlTypeName.DOUBLE))
      

      The RexNodeJsonSerializer uses `typeName`(which is DECIMAL) as the literal's type, as a result, the rel data type is serialized as double, but the value is serialized as a string (in case lost the precision)
      https://github.com/apache/flink/blob/f9b3e0b7bc0432001b4a197539a0712b16e0b33b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/serde/RexNodeJsonSerializer.java#L197

      And then, during the deserialization, according to the JSON, the deserialized literal will assign DOUBLE as the literal type and the broad type of the literal.
      This will cause the comparison failure

      expected: {"kind": "LITERAL", "value": "1.7976931348623157E+308"}
      actual: {"kind": "LITERAL", "value": 1.7976931348623157E+308}
      
      Suggested Fix

      SARG is a special case and can be coped first, and for the rest type, we can use literal.getType().getSqlTypeName() instead of literal.getTypeName().

      // first cope with SARG type
      if (literal.getTypeName() == SARG) {
          serializeSargValue(
              (Sarg<?>) value, literal.getType().getSqlTypeName(), gen, serializerProvider);
      } else {
          serializeLiteralValue(
              value,
              literal.getType().getSqlTypeName(),
              gen,
              serializerProvider);
      }
      

      Attachments

        Issue Links

          Activity

            People

              qingyue Jane Chan
              qingyue Jane Chan
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: