Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-45869 Revisit and Improve Spark Standalone Cluster
  3. SPARK-47934

Ensure trailing slashes in `HistoryServer` URL redirections

    XMLWordPrintableJSON

Details

    Description

      Summary:
      The current implementation of URL redirection in Spark's history web UI does not consistently add trailing slashes to URLs when constructing redirection targets. This inconsistency leads to additional HTTP redirects by Jetty, which increases the load time and reduces the efficiency of the Spark UI.

      Problem Description:
      When constructing redirect URLs, particularly in scenarios where an attempt ID needs to be appended, the system does not ensure that the base URL ends with a slash. This omission results in the generated URL being redirected by Jetty to add a trailing slash, thus causing an unnecessary additional HTTP redirect.

      For example, when the `shouldAppendAttemptId` flag is true, the URL is formed without a trailing slash before the attempt ID is appended, leading to two redirects: one by our logic to add the attempt ID, and another by Jetty to correct the missing slash. 

      Proposed Solution:

      https://github.com/apache/spark/blob/2d0b56c3eac611e743c41d16ea8e439bc8a504e4/core/src/main/scala/org/apache/spark/deploy/history/HistoryServer.scala#L118

      Ensure that all redirect URLs uniformly end with a trailing slash regardless of whether an attempt ID is appended. This can be achieved by modifying the URL construction logic as follows:

      ```scala
      val redirect = if (shouldAppendAttemptId)

      { req.getRequestURI.stripSuffix("/") + "/" + attemptId.get + "/" }

      else

      { req.getRequestURI.stripSuffix("/") + "/" }

      ```
       

      Attachments

        Issue Links

          Activity

            People

              huangzhir huangzhir
              huangzhir huangzhir
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: