Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-39457

Support IPv6-only environment

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.4.0
    • 3.4.0
    • Spark Core

    Description

      Spark doesn't fully work in pure IPV6 environment that doesn't have IPV4 at all. This is an umbrella jira tracking the support of pure IPV6 deployment.

       

      The scope of this JIRA doesn't include ancient resource managers like Mesos(deprecated in Spark) or YARN(because Apache Spark still supports Hadoop 2). We are going to focus on only `Spark Standalone` and `K8s`

       

      K8s IPv4/IPv6 dual-stack Feature reached `Stable` stage at v1.23.

       

      Currently, EKS supports IPv6 with v1.21.13-eks-84b4fe6.

      Attachments

        1.
        		 local*HostName* methods should support IPv6 Sub-task Resolved Dongjoon Hyun
        2.
        		 Improve RpcAddress to add [] to IPv6 if needed Sub-task Resolved Dongjoon Hyun
        3.
        		 SocketAuthServer should respect Java IPv6 options Sub-task Resolved Dongjoon Hyun
        4.
        		 Improve SparkContext to propagate `java.net.preferIPv6Addresses` Sub-task Resolved Dongjoon Hyun
        5.
        		 Support `ipFamilyPolicy` and `ipFamilies` in Driver Service Sub-task Resolved Dongjoon Hyun
        6.
        		 KubernetesExecutorBackend should handle IPv6 hostname Sub-task Resolved Dongjoon Hyun
        7.
        		 Add net-tools to Spark docker files Sub-task Resolved Dongjoon Hyun
        8.
        		 Support IPv6 between JVM and Python Daemon in PySpark Sub-task Resolved Dongjoon Hyun
        9.
        		 Improve YARN client mode to support IPv6 Sub-task Resolved Dongjoon Hyun
        10.
        		 Improve YARN cluster mode to support IPv6 Sub-task Resolved Dongjoon Hyun
        11.
        		 Support IPv6 in WebUI Sub-task Resolved Dongjoon Hyun
        12.
        		 Add build and test documentation on IPv6 Sub-task Resolved Dongjoon Hyun
        13.
        		 Fix UISuite for IPv6 Sub-task Resolved Dongjoon Hyun
        14.
        		 Fix CoarseGrainedSchedulerBackendSuite to handle fast allocations Sub-task Resolved Dongjoon Hyun
        15.
        		 Print `SPARK_LOCAL_(HOSTNAME|IP)` in `build/{mvn|sbt}` Sub-task Resolved Dongjoon Hyun
        16.
        		 Use `Utils.localCanonicalHostName` instead of `localhost` in tests Sub-task Resolved Dongjoon Hyun
        17.
        		 Ivy doesn't work correctly on IPv6-only environment Sub-task Closed Unassigned
        18.
        		 Propagate `java.net.preferIPv6Addresses=true` in SBT tests Sub-task Resolved Dongjoon Hyun
        19.
        		 Support DEFAULT_ARTIFACT_REPOSITORY in check-license Sub-task Resolved Dongjoon Hyun
        20.
        		 LauncherBackendSuite should add java.net.preferIPv6Addresses conf Sub-task Resolved Dongjoon Hyun
        21.
        		 Fix KafkaTestUtils to support IPv6 Sub-task Resolved Dongjoon Hyun
        22.
        		 Make hive-thrift server module passes in IPv6 environment Sub-task Resolved Dongjoon Hyun
        23.
        		 Use localHostNameForURI in UISuite Sub-task Resolved William Hyun
        24.
        		 Fix `test_daemon.py` to support IPv6 Sub-task Resolved Dongjoon Hyun
        25.
        		 Improve logs to show addresses in addition to port Sub-task Resolved Dongjoon Hyun
        26.
        		 Principals in KafkaTestUtils should use canonical host name Sub-task Resolved Jungtaek Lim
        27.
        		 Add `IPv4 and IPv6` section to K8s document Sub-task Resolved Dongjoon Hyun
        28.
        		 IPv6 address mapping can fail with sparse addresses Sub-task Resolved Holden Karau

        Activity

          People

            dongjoon Dongjoon Hyun
            dbtsai DB Tsai
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: