Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-6951

History server slow startup if the event log directory is large

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.3.0
    • 2.4.0
    • Web UI
    • None

    Description

      I started my history server, then navigated to the web UI where I expected to be able to view some completed applications, but the webpage was not available. It turned out that the History Server was not finished parsing all of the event logs in the event log directory that I had specified. I had accumulated a lot of event logs from months of running Spark, so it would have taken a very long time for the History Server to crunch through them all. I purged the event log directory and started from scratch, and the UI loaded immediately.

      We should have a pagination strategy or parse the directory lazily to avoid needing to wait after starting the history server.

      Attachments

        Issue Links

          is duplicated by

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-9123 Spark HistoryServer load logs too slow and can load the latest logs

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-11123 Inprove HistoryServer with multithread to relay logs

          • Major - Major loss of function.
          • Resolved
          is related to

          Umbrella - An overarching type made of sub-tasks SPARK-18085 SPIP: Better History Server scalability for many / large applications

          • Major - Major loss of function.
          • Resolved
          relates to

          New Feature - A new feature of the product, which has yet to be developed. SPARK-11373 Add metrics to the History Server and providers

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-13988 Large history files block new applications from showing up in History UI.

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-18010 Remove unneeded heavy work performed by FsHistoryProvider for building up the application listing UI page

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #20952 [Github] Pull Request #20952 (vanzin)

          Activity

            People

              vanzin Marcelo Masiero Vanzin
              mcheah Matt Cheah
              Votes:
              3 Vote for this issue
              Watchers:
              21 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: