Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-20064

Data cannot be inserted into a dynamic partition

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • None
    • None
    • SQL
    • None

    Description

      Data cannot be inserted into a dynamic partition,The relevant table information is as follows:

      The source table:

      //代码占位符
      create table customer(id string,name string,gender string,state string) partitioned by(country string);
      
      

      The source table data:

       

      //代码占位符
      +--------------+----------------+------------------+-----------------+-------------------+--+
      | customer.id | customer.name | customer.gender | customer.state | customer.country |
      +--------------+----------------+------------------+-----------------+-------------------+--+
      | 1 | gavin | 1 | shanghai | china |
      +--------------+----------------+------------------+-----------------+-------------------+--+
      
      

       

      The target table:

       

      //代码占位符
      create table customer_bak(id string,name string,gender string,state string) partitioned by(country string);
      
      

       

      Now I use dynamic techniques to insert data into the customer_bak table:

      //代码占位符
      set hive.exec.dynamic.partition=true;
      set hive.exec.dynamic.partition.mode=nonstrict;
      insert into table customer_bak partition(country) select id,name,gender,state,country from customer;
      

      The program is always executing and neither success nor failure is shown. The information in the hive log is as follows:

      //代码占位符
      
      2018-06-28T16:01:22,900 INFO [HiveServer2-Background-Pool: Thread-97] ql.Driver: Executing command(queryId=hadoop_20180628160122_f80c5f8e-98a0-4286-bc55-c25efd7a60bc): insert into table customer_bak partition(country) select id,name,gender,state,country from customer
      2018-06-28T16:01:22,901 WARN [HiveServer2-Background-Pool: Thread-97] ql.Driver: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. tez, spark) or using Hive 1.X releases.
      2018-06-28T16:01:22,902 INFO [HiveServer2-Background-Pool: Thread-97] ql.Driver: WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. tez, spark) or using Hive 1.X releases.
      2018-06-28T16:01:22,902 INFO [HiveServer2-Background-Pool: Thread-97] ql.Driver: Query ID = hadoop_20180628160122_f80c5f8e-98a0-4286-bc55-c25efd7a60bc
      2018-06-28T16:01:22,903 INFO [HiveServer2-Background-Pool: Thread-97] ql.Driver: Total jobs = 3
      2018-06-28T16:01:22,904 INFO [HiveServer2-Background-Pool: Thread-97] ql.Driver: Launching Job 1 out of 3
      2018-06-28T16:01:22,907 INFO [HiveServer2-Background-Pool: Thread-97] ql.Driver: Starting task [Stage-1:MAPRED] in serial mode
      2018-06-28T16:01:22,908 INFO [HiveServer2-Background-Pool: Thread-97] exec.Task: Number of reduce tasks is set to 0 since there's no reduce operator
      2018-06-28T16:01:22,929 INFO [HiveServer2-Background-Pool: Thread-97] ql.Context: New scratch dir is hdfs://hadoop-cluster/tmp/hive/hadoop/07fd131c-6590-4081-b7ed-2f0ae5b16551/hive_2018-06-28_16-01-22_402_5116918946052746369-3
      2018-06-28T16:01:22,942 INFO [HiveServer2-Background-Pool: Thread-97] mr.ExecDriver: Using org.apache.hadoop.hive.ql.io.CombineHiveInputFormat
      2018-06-28T16:01:22,943 INFO [HiveServer2-Background-Pool: Thread-97] exec.Utilities: Processing alias customer
      2018-06-28T16:01:22,943 INFO [HiveServer2-Background-Pool: Thread-97] exec.Utilities: Adding 2 inputs; the first input is hdfs://hadoop-cluster/user/hive/warehouse/customer/country=china
      2018-06-28T16:01:22,943 INFO [HiveServer2-Background-Pool: Thread-97] exec.Utilities: Content Summary not cached for hdfs://hadoop-cluster/user/hive/warehouse/customer/country=china
      2018-06-28T16:01:22,945 INFO [HiveServer2-Background-Pool: Thread-97] exec.Utilities: Content Summary not cached for hdfs://hadoop-cluster/user/hive/warehouse/customer/country=usa
      2018-06-28T16:01:22,978 INFO [HiveServer2-Background-Pool: Thread-97] exec.Utilities: Changed input file hdfs://hadoop-cluster/user/hive/warehouse/customer/country=usa to empty file hdfs://hadoop-cluster/tmp/hive/hadoop/07fd131c-6590-4081-b7ed-2f0ae5b16551/hive_2018-06-28_16-01-22_402_5116918946052746369-3/-mr-10003/408baabf-a30d-4c00-a196-70e14e172036 (false)
      2018-06-28T16:01:22,979 INFO [HiveServer2-Background-Pool: Thread-97] ql.Context: New scratch dir is hdfs://hadoop-cluster/tmp/hive/hadoop/07fd131c-6590-4081-b7ed-2f0ae5b16551/hive_2018-06-28_16-01-22_402_5116918946052746369-3
      2018-06-28T16:01:22,996 INFO [HiveServer2-Background-Pool: Thread-97] exec.SerializationUtilities: Serializing MapWork using kryo
      2018-06-28T16:01:23,055 INFO [HiveServer2-Background-Pool: Thread-97] exec.Utilities: Serialized plan (via FILE) - name: null size: 4.86KB
      2018-06-28T16:01:29,318 INFO [HiveServer2-Background-Pool: Thread-97] fs.FSStatsPublisher: created : hdfs://hadoop-cluster/user/hive/warehouse/customer_bak/.hive-staging_hive_2018-06-28_16-01-22_402_5116918946052746369-1/-ext-10001
      2018-06-28T16:01:29,395 INFO [HiveServer2-Background-Pool: Thread-97] exec.Utilities: PLAN PATH = hdfs://hadoop-cluster/tmp/hive/hadoop/07fd131c-6590-4081-b7ed-2f0ae5b16551/hive_2018-06-28_16-01-22_402_5116918946052746369-3/-mr-10004/2cb63e68-2c77-4554-817a-0e25d700cf0f/map.xml
      2018-06-28T16:01:29,395 INFO [HiveServer2-Background-Pool: Thread-97] exec.Utilities: PLAN PATH = hdfs://hadoop-cluster/tmp/hive/hadoop/07fd131c-6590-4081-b7ed-2f0ae5b16551/hive_2018-06-28_16-01-22_402_5116918946052746369-3/-mr-10004/2cb63e68-2c77-4554-817a-0e25d700cf0f/reduce.xml
      2018-06-28T16:01:29,398 INFO [HiveServer2-Background-Pool: Thread-97] exec.Utilities: No plan file found: hdfs://hadoop-cluster/tmp/hive/hadoop/07fd131c-6590-4081-b7ed-2f0ae5b16551/hive_2018-06-28_16-01-22_402_5116918946052746369-3/-mr-10004/2cb63e68-2c77-4554-817a-0e25d700cf0f/reduce.xml
      2018-06-28T16:01:29,406 INFO [HiveServer2-Background-Pool: Thread-97] client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
      2018-06-28T16:01:29,413 WARN [HiveServer2-Background-Pool: Thread-97] mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
      2018-06-28T16:01:31,721 INFO [HiveServer2-Background-Pool: Thread-97] exec.Utilities: PLAN PATH = hdfs://hadoop-cluster/tmp/hive/hadoop/07fd131c-6590-4081-b7ed-2f0ae5b16551/hive_2018-06-28_16-01-22_402_5116918946052746369-3/-mr-10004/2cb63e68-2c77-4554-817a-0e25d700cf0f/map.xml
      2018-06-28T16:01:31,722 INFO [HiveServer2-Background-Pool: Thread-97] io.CombineHiveInputFormat: Total number of paths: 2, launching 1 threads to check non-combinable ones.
      2018-06-28T16:01:31,724 INFO [HiveServer2-Background-Pool: Thread-97] io.CombineHiveInputFormat: CombineHiveInputSplit creating pool for hdfs://hadoop-cluster/user/hive/warehouse/customer/country=china; using filter path hdfs://hadoop-cluster/user/hive/warehouse/customer/country=china
      2018-06-28T16:01:31,725 INFO [HiveServer2-Background-Pool: Thread-97] io.CombineHiveInputFormat: CombineHiveInputSplit: pool is already created for hdfs://hadoop-cluster/tmp/hive/hadoop/07fd131c-6590-4081-b7ed-2f0ae5b16551/hive_2018-06-28_16-01-22_402_5116918946052746369-3/-mr-10003/408baabf-a30d-4c00-a196-70e14e172036; using filter path hdfs://hadoop-cluster/tmp/hive/hadoop/07fd131c-6590-4081-b7ed-2f0ae5b16551/hive_2018-06-28_16-01-22_402_5116918946052746369-3/-mr-10003/408baabf-a30d-4c00-a196-70e14e172036
      2018-06-28T16:01:31,738 INFO [HiveServer2-Background-Pool: Thread-97] input.FileInputFormat: Total input paths to process : 2
      2018-06-28T16:01:31,742 INFO [HiveServer2-Background-Pool: Thread-97] input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 3, size left: 0
      2018-06-28T16:01:31,742 INFO [HiveServer2-Background-Pool: Thread-97] io.CombineHiveInputFormat: number of splits 1
      2018-06-28T16:01:31,743 INFO [HiveServer2-Background-Pool: Thread-97] io.CombineHiveInputFormat: Number of all splits 1
      2018-06-28T16:01:31,853 INFO [HiveServer2-Background-Pool: Thread-97] mapreduce.JobSubmitter: number of splits:1
      2018-06-28T16:01:31,982 INFO [HiveServer2-Background-Pool: Thread-97] mapreduce.JobSubmitter: Submitting tokens for job: job_1530599349924_0007
      2018-06-28T16:01:32,220 INFO [HiveServer2-Background-Pool: Thread-97] impl.YarnClientImpl: Submitted application application_1530599349924_0007
      2018-06-28T16:01:32,226 INFO [HiveServer2-Background-Pool: Thread-97] mapreduce.Job: The url to track the job: http://cluster3:8088/proxy/application_1530599349924_0007/
      2018-06-28T16:01:32,227 INFO [HiveServer2-Background-Pool: Thread-97] exec.Task: Starting Job = job_1530599349924_0007, Tracking URL = http://cluster3:8088/proxy/application_1530599349924_0007/
      2018-06-28T16:01:32,227 INFO [HiveServer2-Background-Pool: Thread-97] exec.Task: Kill Command = /home/hadoop/app/hadoop_2.6/bin/hadoop job -kill job_1530599349924_0007
      2018-06-28T16:01:56,835 INFO [HiveServer2-Background-Pool: Thread-97] exec.Task: Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
      2018-06-28T16:01:56,933 WARN [HiveServer2-Background-Pool: Thread-97] mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
      2018-06-28T16:01:56,934 INFO [HiveServer2-Background-Pool: Thread-97] exec.Task: 2018-06-28 16:01:56,933 Stage-1 map = 0%, reduce = 0%
      2018-06-28T16:02:05,922 INFO [HiveServer2-Handler-Pool: Thread-30] conf.HiveConf: Using the default value passed in for log id: 07fd131c-6590-4081-b7ed-2f0ae5b16551
      2018-06-28T16:02:05,922 INFO [HiveServer2-Handler-Pool: Thread-30] session.SessionState: Updating thread name to 07fd131c-6590-4081-b7ed-2f0ae5b16551 HiveServer2-Handler-Pool: Thread-30
      2018-06-28T16:02:05,924 INFO [07fd131c-6590-4081-b7ed-2f0ae5b16551 HiveServer2-Handler-Pool: Thread-30] conf.HiveConf: Using the default value passed in for log id: 07fd131c-6590-4081-b7ed-2f0ae5b16551
      2018-06-28T16:02:05,925 INFO [HiveServer2-Handler-Pool: Thread-30] session.SessionState: Resetting thread name to HiveServer2-Handler-Pool: Thread-30
      2018-06-28T16:02:35,131 INFO [HiveServer2-Background-Pool: Thread-97] exec.Task: 2018-06-28 16:02:35,130 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 2.82 sec
      2018-06-28T16:02:39,527 INFO [HiveServer2-Background-Pool: Thread-97] exec.Task: MapReduce Total cumulative CPU time: 2 seconds 820 msec
      2018-06-28T16:02:39,607 INFO [HiveServer2-Background-Pool: Thread-97] exec.Task: Ended Job = job_1530599349924_0007
      2018-06-28T16:02:39,643 INFO [HiveServer2-Background-Pool: Thread-97] exec.FileSinkOperator: Moving tmp dir: hdfs://hadoop-cluster/user/hive/warehouse/customer_bak/.hive-staging_hive_2018-06-28_16-01-22_402_5116918946052746369-1/_tmp.-ext-10002 to: hdfs://hadoop-cluster/user/hive/warehouse/customer_bak/.hive-staging_hive_2018-06-28_16-01-22_402_5116918946052746369-1/-ext-10002
      2018-06-28T16:02:39,669 INFO [HiveServer2-Background-Pool: Thread-97] ql.Driver: Starting task [Stage-7:CONDITIONAL] in serial mode
      2018-06-28T16:02:39,688 INFO [HiveServer2-Background-Pool: Thread-97] exec.Task: Stage-4 is selected by condition resolver.
      2018-06-28T16:02:39,689 INFO [HiveServer2-Background-Pool: Thread-97] exec.Task: Stage-3 is filtered out by condition resolver.
      2018-06-28T16:02:39,689 INFO [HiveServer2-Background-Pool: Thread-97] exec.Task: Stage-5 is filtered out by condition resolver.
      2018-06-28T16:02:39,690 INFO [HiveServer2-Background-Pool: Thread-97] ql.Driver: Starting task [Stage-4:MOVE] in serial mode
      2018-06-28T16:02:39,691 INFO [HiveServer2-Background-Pool: Thread-97] exec.Task: Moving data to directory hdfs://hadoop-cluster/user/hive/warehouse/customer_bak/.hive-staging_hive_2018-06-28_16-01-22_402_5116918946052746369-1/-ext-10000 from hdfs://hadoop-cluster/user/hive/warehouse/customer_bak/.hive-staging_hive_2018-06-28_16-01-22_402_5116918946052746369-1/-ext-10002
      2018-06-28T16:02:39,785 INFO [HiveServer2-Background-Pool: Thread-97] ql.Driver: Starting task [Stage-0:MOVE] in serial mode
      2018-06-28T16:02:39,786 INFO [HiveServer2-Background-Pool: Thread-97] exec.Task: Loading data to table default.customer_bak partition (country=null) from hdfs://hadoop-cluster/user/hive/warehouse/customer_bak/.hive-staging_hive_2018-06-28_16-01-22_402_5116918946052746369-1/-ext-10000
      2018-06-28T16:02:39,871 INFO [HiveServer2-Background-Pool: Thread-97] exec.MoveTask: Partition is: {country=null}
      2018-06-28T16:02:39,879 INFO [HiveServer2-Background-Pool: Thread-97] exec.Task:
      

      Why doesn't the program continue?

       

       

      Attachments

        Activity

          People

            Unassigned Unassigned
            gavin_shi gavin_shi
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: