Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21348

Spark history Server load too slow when eventlog is large

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Duplicate
    • 2.1.0
    • None
    • Web UI
    • None

    Description

      There is a eventlog file in hdfs as below:
      ```
      rwxrwx-- 3 root supergroup 9634238709 2017-07-08 04:37 hdfs://xxxxx/shared/spark-logs/xxxxx/xxxx.lz4
      ```

      and i start a spark history server to loading that eventlog file,but it is too slow to get spark web ui through that spark history server.

      it has requested for several hours but nothing about spark application web ui appears in the browser

      And the cpu usage of spark history server is 105.3%
      total used-memory in host is 9G

      The host has 48 Cores+130G Memory,enough Computing resources for spark history server.

      Attachments

        Issue Links

          Blocked

          Umbrella - An overarching type made of sub-tasks SPARK-18085 SPIP: Better History Server scalability for many / large applications

          • Major - Major loss of function.
          • Resolved
          duplicates

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-20582 Speed up the restart of HistoryServer using ApplicationAttemptInfo checkpointing

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Umbrella - An overarching type made of sub-tasks SPARK-18085 SPIP: Better History Server scalability for many / large applications

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              duyanghao duyanghao
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: