Uploaded image for project: 'Bahir (Retired)'
  1. Bahir (Retired)
  2. BAHIR-276

flink redis sink bug fix and add pipeline support with at least once semantics

    XMLWordPrintableJSON

Details

    Description

      I've found some bugs and improvements which I would like to work on it in flink-connector-redis.

       

      Bug:

      • when redis container call open(), it will get a Jedis resource for `echo` to validate redis connection, but this resource never close.

       

      Improvement:

      • redis container use many try..catch..finally(release jedis) block, which can be replaced by `try with resource`.
      • redis container use Jedis Pool to provide Jedis resource instance, every element send into sink will get a Jedis instance, do redis command, and then close Jedis instance. When upstream produces tons of data (such as 10w ops), it will suffer extremely performance degradation. With Jedis pipeline, the performance will be improved outstandingly (in our test cases, `hincrBy` can be improved from 5w+ ops to  60w+ ops), and still guarantees flink `at least once` semantics.

       

      I've already finished the improvement above and used in our own production environment. I would like to contribute it to the bahir project.

      Attachments

        Activity

          People

            Unassigned Unassigned
            rinkako rinkako
            Votes:
            2 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: