Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25376

Scenarios we should handle but missed in 2.4 for barrier execution mode

    XMLWordPrintableJSON

Details

    • Story
    • Status: Resolved
    • Major
    • Resolution: Incomplete
    • 2.4.0
    • None
    • Spark Core

    Description

      irashid You mentioned that there are couple scenarios we should handle in barrier execution mode but we didn't in 2.4. Could you elaborate here?

      One scenario we are aware of is that speculation is not supported by barrier mode. Hence a barrier mode might hang in case of hardware issues on one node. I don't have a good proposal here except letting users set a timeout for the barrier stage. Would like to hear your thoughts.

      You also mentioned multi-tenancy issues. Could you say more?

      cc: jiangxb1987

      Attachments

        Activity

          People

            Unassigned Unassigned
            mengxr Xiangrui Meng
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: