Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28778

Shuffle jobs fail due to incorrect advertised address when running in virtual network

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.2.3, 2.3.0, 2.4.3
    • 2.4.5, 3.0.0
    • Mesos

    Description

      When shuffle jobs are launched by Mesos in a virtual network, Mesos scheduler sets executor --hostname parameter to 0.0.0.0 in the case when spark.mesos.network.name is provided. This makes executors use 0.0.0.0 as their advertised address and, in the presence of shuffle, executors fail to fetch shuffle blocks from each other using 0.0.0.0 as the origin. When a virtual network is used the hostname or IP address is not known upfront and assigned to a container at its start time so the executor process needs to advertise the correct dynamically assigned address to be reachable by other executors.

       

      The bug described above prevents Mesos users from running any jobs which involve shuffle due to the inability of executors to fetch shuffle blocks because of incorrect advertised address when virtual network is used.

      Attachments

        Activity

          People

            anton.kirillov Anton Kirillov
            anton.kirillov Anton Kirillov
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: