Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25357

Add metadata to SparkPlanInfo to dump more information like file path to event log

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 2.3.1
    • 2.3.2, 2.4.0
    • SQL
    • None

    Description

      Field metadata removed from SparkPlanInfo in SPARK-17701. Corresponding, this field was also removed from event SparkListenerSQLExecutionStart in Spark event log. If we want to analyze event log to get some fields which wider than 100 (e.g the Location or ReadSchema of FileScan), they are abbreviated in simpleString of SparkPlanInfo JSON or physicalPlanDescription JSON.

      Before 2.3, the fragment of SparkListenerSQLExecutionStart in event log (It contains the metadata field):

      Location: InMemoryFileIndex[hdfs://hercules/sys/edw/prs_idm/idm_cbt_am_t/cbt/cbt_acct_prfl_info/snapshot/dt..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<snpsht_start_dt:date,snpsht_end_dt:date,am_ntlogin_name:string,am_first_name:string,am_las...","children":[],"metadata":{"Location":"InMemoryFileIndex[hdfs://hercules/sys/edw/prs_idm/idm_cbt_am_t/cbt/cbt_acct_prfl_info/snapshot/dt=20180904]","ReadSchema":"struct<snpsht_start_dt:date,snpsht_end_dt:date,am_ntlogin_name:string,am_first_name:string,am_last_name:string,isg_name:string,acct_isg_stat_desc:string,prmry_user_slctd_id:string,prmry_orcl_id:bigint,acct_cmpny_bsns_lcns_num:string,acct_slctd_id:string,acct_orcl_id:bigint,acct_cmpny_name:string,acct_cmpny_region_txt:string,acct_cmpny_prvnc_txt:string,acct_cmpny_addr_txt:string,acct_type_seg:string,p4_acct_ind:tinyint,i320_acct_ind:tinyint,i463_acct_ind:tinyint,i319_acct_ind:tinyint,acct_cntry:string,acct_stat:string,acct_club_ind:string,acct_src_bd_name:string,acct_prmry_bsns_vrtcl_desc:string,acct_minor_bsns_vrtcl_desc:string,acct_src_desc:string,acct_pre_ams_id:bigint,src_last_mdfd_dt:date,src_last_mdfd_tm:string,CRE_DATE:date,CRE_USER:string,UPD_DATE:timestamp,UPD_USER:string>"

      So I add this field back to SparkPlanInfo class. Then it will log out the meta data to event log. Intact information in event log is very useful for offline job analysis.

      Attachments

        Activity

          People

            cltlfcjin Lantao Jin
            cltlfcjin Lantao Jin
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: