Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-18085

SPIP: Better History Server scalability for many / large applications

    XMLWordPrintableJSON

Details

    • Umbrella
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.0.0
    • 2.3.0
    • Spark Core, Web UI

    Description

      It's a known fact that the History Server currently has some annoying issues when serving lots of applications, and when serving large applications.

      I'm filing this umbrella to track work related to addressing those issues. I'll be attaching a document shortly describing the issues and suggesting a path to how to solve them.

      Attachments

        1. screenshot-1.png
          304 kB
          Jincheng Sun
        2. screenshot-2.png
          268 kB
          Jincheng Sun
        3. spark_hs_next_gen.pdf
          99 kB
          Marcelo Masiero Vanzin

        Issue Links

          Blocked

          Bug - A problem which impairs or prevents the functions of the product. SPARK-21348 Spark history Server load too slow when eventlog is large

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          blocks

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-20656 Incremental parsing of event logs in SHS

          • Major - Major loss of function.
          • Resolved
          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. SPARK-20205 DAGScheduler posts SparkListenerStageSubmitted before updating stage

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-20342 DAGScheduler sends SparkListenerTaskEnd before updating task's accumulators

          • Major - Major loss of function.
          • Resolved
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. SPARK-21348 Spark history Server load too slow when eventlog is large

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-20582 Speed up the restart of HistoryServer using ApplicationAttemptInfo checkpointing

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-21254 History UI: Taking over 1 minute for initial page display

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-19814 Spark History Server Out Of Memory / Extreme GC

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. SPARK-20421 Mark JobProgressListener (and related classes) as deprecated

          • Major - Major loss of function.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-6951 History server slow startup if the event log directory is large

          • Major - Major loss of function.
          • Resolved
          1.
          		 Key-value store abstraction and implementation for storing application data Sub-task Resolved Marcelo Masiero Vanzin
          2.
          		 Use key-value store to keep History Server application listing Sub-task Resolved Marcelo Masiero Vanzin
          3.
          		 Implement listener for saving application status data in key-value store Sub-task Resolved Marcelo Masiero Vanzin
          4.
          		 Hook up Spark UI to the new key-value store backend Sub-task Resolved Marcelo Masiero Vanzin
          5.
          		 Make Environment page use new app state store Sub-task Resolved Marcelo Masiero Vanzin
          6.
          		 Make Executors page use new app state store Sub-task Resolved Marcelo Masiero Vanzin
          7.
          		 Make the Storage page use new app state store Sub-task Resolved Marcelo Masiero Vanzin
          8.
          		 Make Jobs and Stages pages use the new app state store Sub-task Resolved Marcelo Masiero Vanzin
          9.
          		 Simplify REST API class hierarchy Sub-task Resolved Marcelo Masiero Vanzin
          10.
          		 Remove JobProgressListener (and other unneeded classes) Sub-task Resolved Marcelo Masiero Vanzin
          11.
          		 Speed up the new app state listener Sub-task Resolved Unassigned
          12.
          		 Make SQL UI use new app state store Sub-task Resolved Marcelo Masiero Vanzin
          13.
          		 Add auto-cleanup of old elements to the new app state store Sub-task Resolved Marcelo Masiero Vanzin
          14.
          		 Add controls for how much disk the SHS can use Sub-task Resolved Marcelo Masiero Vanzin
          15.
          		 In-memory key-value store implementation Sub-task Resolved Marcelo Masiero Vanzin
          16.
          		 Speed up Stage page Sub-task Resolved Marcelo Masiero Vanzin
          17.
          		 Remove stale applications from SHS listing Sub-task Resolved Marcelo Masiero Vanzin
          18.
          		 Move kvstore package to util.kvstore, add annotations Sub-task Resolved Marcelo Masiero Vanzin

          Activity

            People

              vanzin Marcelo Masiero Vanzin
              vanzin Marcelo Masiero Vanzin
              Marcelo Masiero Vanzin Marcelo Masiero Vanzin
              Votes:
              9 Vote for this issue
              Watchers:
              65 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: