Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-33961

Hybrid Shuffle may hang when exclusive buffers per channel is set to 0

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Invalid
    • None
    • None
    • Runtime / Network
    • None

    Description

      I found that the Hybrid Shuffle without enabling new mode may hang when exclusive-buffers-per-channel is set to 0. It can be reproduced by adding the following test into `HybridShuffleITCase.java` and running it.

      @RepeatedTest(10)
      void testHybridFullExchangesWithNonBuffersPerChannel() throws Exception {
          final int numRecordsToSend = 10000;
          Configuration configuration = configureHybridOptions(getConfiguration(), false);
          configuration.set(
                  NettyShuffleEnvironmentOptions.NETWORK_HYBRID_SHUFFLE_ENABLE_NEW_MODE, false);
          configuration.set(NETWORK_BUFFERS_PER_CHANNEL, 0);
          JobGraph jobGraph = createJobGraph(numRecordsToSend, false, configuration);
          executeJob(jobGraph, configuration, numRecordsToSend);
      } 

      Attachments

        Activity

          People

            Unassigned Unassigned
            Jiang Xin Jiang Xin
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: