Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-16686

[State TTL] Make user class loader available in native RocksDB compaction thread

    XMLWordPrintableJSON

Details

    Description

      The issue is initially reported here.

      The problem is that the java code of Flink compaction filter is called from RocksDB native C++ code. It is called in the context of the native compaction thread. RocksDB has utilities to create java Thread context for the Flink java callback. Presumably, the Java thread context class loader is not set at all and if it is queried then it produces NullPointerException.

      The provided report enabled a list state with TTL. The compaction filter has to deserialise elements to check expiration. The deserialiser relies on Kryo which queries the thread context class loader which is expected to be the user class loader of the task but turns out to be null.

      We should investigate how to pass the user class loader to the compaction thread of the list state with TTL.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              azagrebin Andrey Zagrebin
              Votes:
              1 Vote for this issue
              Watchers:
              26 Start watching this issue

              Dates

                Created:
                Updated: