Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-16333

Excessive Spark history event/json data size (5GB each)

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 2.0.0
    • None
    • Spark Core

    • this is seen on both x86 (Intel(R) Xeon(R), E5-2699 ) and ppc platform (Habanero, Model: 8348-21C), Red Hat Enterprise Linux Server release 7.2 (Maipo)., Spark2.0.0-preview (May-24, 2016 build)

    Description

      With Spark2.0.0-preview (May-24 build), the history event data (the json file), that is generated for each Spark application run (see below), can be as big as 5GB (instead of 14 MB for exactly the same application run and the same input data of 1TB under Spark1.6.1)

      rwxrwx-- 1 root root 5.3G Jun 30 09:39 app-20160630091959-0000
      rwxrwx-- 1 root root 5.3G Jun 30 09:56 app-20160630094213-0000
      rwxrwx-- 1 root root 5.3G Jun 30 10:13 app-20160630095856-0000
      rwxrwx-- 1 root root 5.3G Jun 30 10:30 app-20160630101556-0000

      The test is done with Sparkbench V2, SQL RDD (see github: https://github.com/SparkTC/spark-bench)

      Attachments

        Issue Links

          duplicates

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-20084 Remove internal.metrics.updatedBlockStatuses accumulator from history files

          • Major - Major loss of function.
          • Resolved
          is depended upon by

          Bug - A problem which impairs or prevents the functions of the product. SPARK-19111 S3 Mesos history upload fails silently if too large

          • Major - Major loss of function.
          • Resolved
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. SPARK-16332 the history server of spark2.0-preview (may-24 build) consumes more than 1000% cpu

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-19316 Spark event logs are huge compared to 1.5.2

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #16714 [Github] Pull Request #16714 (jisookim0513)

          Activity

            People

              Unassigned Unassigned
              petergangliu Peter Liu
              Votes:
              3 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: