Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-7254

sqoop on hadoop3.1 doesn't work

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Not A Bug
    • 3.1.0
    • None
    • job submission
    • None

    Description

          I have a mysql table called admin with "linyouquan". And I want to import data from mysql to local filesystem. I chose sqoop for this thing. sqoop on hadoop2.6 works, but sqoop on hadoop3.1 doesn't work.

          The following is my operation and error information of sqoop on hadoop3.1.

      1. my operation
      export HADOOP_HOME=/home/linyouquan/hadoop3-hadoop-pack
      
      sqoop import "-Dorg.apache.sqoop.splitter.allow_text_splitter=true" --connect jdbc:mysql://dbHost:dbPort/dbName --username dbUser --password dbPasswd --table admin --target-dir sqoop_import_user
      
      1. error information of sqoop on hadoop3.1
      Warning: /home/linyouquan/yarn/YARN/sqoop-1.4.7.bin__hadoop-2.6.0/bin/../../hbase does not exist! HBase imports will fail.
      Please set $HBASE_HOME to the root of your HBase installation.
      Warning: /home/linyouquan/yarn/YARN/sqoop-1.4.7.bin__hadoop-2.6.0/bin/../../hcatalog does not exist! HCatalog jobs will fail.
      Please set $HCAT_HOME to the root of your HCatalog installation.
      Warning: /home/linyouquan/yarn/YARN/sqoop-1.4.7.bin__hadoop-2.6.0/bin/../../accumulo does not exist! Accumulo imports will fail.
      Please set $ACCUMULO_HOME to the root of your Accumulo installation.
      Warning: /home/linyouquan/yarn/YARN/sqoop-1.4.7.bin__hadoop-2.6.0/bin/../../zookeeper does not exist! Accumulo imports will fail.
      Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
      2019-12-24 18:29:55,473 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
      2019-12-24 18:29:55,509 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
      2019-12-24 18:29:55,623 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
      2019-12-24 18:29:55,623 INFO tool.CodeGenTool: Beginning code generation
      2019-12-24 18:29:56,087 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `admin` AS t LIMIT 1
      2019-12-24 18:29:56,110 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `admin` AS t LIMIT 1
      2019-12-24 18:29:56,116 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /home/linyouquan/hadoop3-hadoop-pack
      Note: /tmp/sqoop-linyouquan/compile/cf7897369f7ede4babaf39adfd2e55aa/admin.java uses or overrides a deprecated API.
      Note: Recompile with -Xlint:deprecation for details.
      2019-12-24 18:29:57,451 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-linyouquan/compile/cf7897369f7ede4babaf39adfd2e55aa/admin.jar
      2019-12-24 18:29:57,464 WARN manager.MySQLManager: It looks like you are importing from mysql.
      2019-12-24 18:29:57,464 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
      2019-12-24 18:29:57,464 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
      2019-12-24 18:29:57,464 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
      2019-12-24 18:29:57,468 INFO mapreduce.ImportJobBase: Beginning import of admin
      2019-12-24 18:29:57,469 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
      2019-12-24 18:29:57,639 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
      2019-12-24 18:29:57,770 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
      2019-12-24 18:29:58,051 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
      2019-12-24 18:29:58,117 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
      2019-12-24 18:29:58,117 INFO impl.MetricsSystemImpl: JobTracker metrics system started
      2019-12-24 18:29:58,244 INFO db.DBInputFormat: Using read commited transaction isolation
      2019-12-24 18:29:58,260 INFO mapreduce.JobSubmitter: number of splits:1
      2019-12-24 18:29:58,378 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1105755779_0001
      2019-12-24 18:29:58,380 INFO mapreduce.JobSubmitter: Executing with tokens: []
      2019-12-24 18:29:58,599 INFO mapred.LocalDistributedCacheManager: Creating symlink: /tmp/hadoop-linyouquan/mapred/local/1577183398474/libjars <- /home/linyouquan/yarn/YARN/libjars/*
      2019-12-24 18:29:58,601 WARN fs.FileUtil: Command 'ln -s /tmp/hadoop-linyouquan/mapred/local/1577183398474/libjars /home/linyouquan/yarn/YARN/libjars/*' failed 1 with: ln: creating symbolic link `/home/linyouquan/yarn/YARN/libjars/*': No such file or directory2019-12-24 18:29:58,601 WARN mapred.LocalDistributedCacheManager: Failed to create symlink: /tmp/hadoop-linyouquan/mapred/local/1577183398474/libjars <- /home/linyouquan/yarn/YARN/libjars/*
      2019-12-24 18:29:58,602 INFO mapred.LocalDistributedCacheManager: Localized file:/tmp/hadoop/mapred/staging/linyouquan1105755779/.staging/job_local1105755779_0001/libjars as file:/tmp/hadoop-linyouquan/mapred/local/1577183398474/libjars
      2019-12-24 18:29:58,673 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
      2019-12-24 18:29:58,674 INFO mapreduce.Job: Running job: job_local1105755779_0001
      2019-12-24 18:29:58,674 INFO mapred.LocalJobRunner: OutputCommitter set in config null
      2019-12-24 18:29:58,682 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
      2019-12-24 18:29:58,682 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
      2019-12-24 18:29:58,682 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
      2019-12-24 18:29:58,701 INFO mapred.LocalJobRunner: Waiting for map tasks
      2019-12-24 18:29:58,701 INFO mapred.LocalJobRunner: Starting task: attempt_local1105755779_0001_m_000000_0
      2019-12-24 18:29:58,725 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
      2019-12-24 18:29:58,725 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
      2019-12-24 18:29:58,739 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
      2019-12-24 18:29:58,744 INFO db.DBInputFormat: Using read commited transaction isolation
      2019-12-24 18:29:58,748 INFO mapred.MapTask: Processing split: 1=1 AND 1=1
      2019-12-24 18:29:58,754 INFO mapred.LocalJobRunner: map task executor complete.
      2019-12-24 18:29:58,755 WARN mapred.LocalJobRunner: job_local1105755779_0001
      java.lang.Exception: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class admin not found
      	at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
      	at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552)
      Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class admin not found
      	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2597)
      	at org.apache.sqoop.mapreduce.db.DBConfiguration.getInputClass(DBConfiguration.java:403)
      	at org.apache.sqoop.mapreduce.db.DataDrivenDBInputFormat.createDBRecordReader(DataDrivenDBInputFormat.java:270)
      	at org.apache.sqoop.mapreduce.db.DBInputFormat.createRecordReader(DBInputFormat.java:266)
      	at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.<init>(MapTask.java:534)
      	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:777)
      	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348)
      	at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: java.lang.ClassNotFoundException: Class admin not found
      	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2501)
      	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2595)
      	... 12 more
      2019-12-24 18:29:59,678 INFO mapreduce.Job: Job job_local1105755779_0001 running in uber mode : false
      2019-12-24 18:29:59,679 INFO mapreduce.Job:  map 0% reduce 0%
      2019-12-24 18:29:59,682 INFO mapreduce.Job: Job job_local1105755779_0001 failed with state FAILED due to: NA
      2019-12-24 18:29:59,689 INFO mapreduce.Job: Counters: 0
      2019-12-24 18:29:59,693 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
      2019-12-24 18:29:59,695 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 1.913 seconds (0 bytes/sec)
      2019-12-24 18:29:59,695 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
      2019-12-24 18:29:59,695 INFO mapreduce.ImportJobBase: Retrieved 0 records.
      2019-12-24 18:29:59,695 ERROR tool.ImportTool: Import failed: Import job failed!
      

      Attachments

        Activity

          People

            Unassigned Unassigned
            Youquan Lin Youquan Lin
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: