Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-8691

AMRMClient unregisterApplicationMaster Api's appMessage should have a maximum size

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Duplicate
    • 2.7.3
    • 3.0.0-alpha4
    • resourcemanager
    • None

    Description

      SparkSQL AM Codegen ERROR，then call unregister AM API and send the error message to RM, RM receive the AM state and update to RMStateStore. The  Codegen error message maybe is huge, (Our case is about 200MB). If the RMStateStore is ZKRMStateStore, it causes the same exception as YARN-6125, but YARN-6125 doesn't cover the unregisterApplicationMaster's message cut.

       

      SparkSQL Codegen error message show below:

      18/08/18 08:34:54 ERROR codegen.CodeGenerator: failed to compile: org.codehaus.janino.JaninoRuntimeException: Constant pool has grown past JVM limit of 0xFFFF
      /* 001 */ public java.lang.Object generate(Object[] references)

      { /* 002 */ return new SpecificSafeProjection(references); /* 003 */ }

      /* 004 */
      /* 005 */ class SpecificSafeProjection extends org.apache.spark.sql.catalyst.expressions.codegen.BaseProjection {
      ..........

      about 2 million lines.

      ..........

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. YARN-6967 Limit application attempt's diagnostic message size thoroughly

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-25174 ApplicationMaster suspends when unregistering itself from RM with extreme large diagnostic message

          • Major - Major loss of function.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. YARN-7115 Move BoundedAppender to org.hadoop.yarn.util pacakge

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. YARN-6125 The application attempt's diagnostic message should have a maximum size

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Activity

            People

              caiyicong Yicong Cai
              caiyicong Yicong Cai
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: