Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-22693

Modify fs.defaultFS. The location of the new table does not change. It is still the value of fs.defaultFS before modification

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Not A Bug
    • 3.1.0
    • None
    • Hive
    • None
    • HDP: 3.1.4.0

      Hadoop: 3.1.1 

      Hive: 3.1.0

      HBase: 2.0.2

      Tez:0.9.1

    • Hide
      Phenomenon caused by hadoop core-site.xml, configured as final parameter:
      <property>
              <name>fs.defaultFS</name>
              <value>hdfs://master:8020&lt;/value>
              <final>true</final>
          </property>
      Show
      Phenomenon caused by hadoop core-site.xml, configured as final parameter: <property>         <name>fs.defaultFS</name>         <value> hdfs://master:8020&lt;/value >         <final>true</final>     </property>

    Description

      Change the hive-site. xml configuration item fs.defaultfs, or set fs.defaultfs =alluxio://master:19998 in beeline, without effect, such as create table dual(data string), table LOCATION is still the previous fs.defaultFS value hdfs://master:8020. By rights, FS in LOCATION should be the modified alluxio://master:19998

       

      Example log:

      [root@master ~]# beeline
      SLF4J: Class path contains multiple SLF4J bindings.
      SLF4J: Found binding in [jar:file:/usr/hdp/3.1.4.0-315/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: Found binding in [jar:file:/usr/hdp/3.1.4.0-315/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
      SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
      20/01/04 20:09:58 [main]: WARN conf.Configuration: file:/etc/hive/3.1.4.0-315/0/hive-site.xml:an attempt to override final parameter: fs.defaultFS; Ignoring.
      Connecting to jdbc:hive2://master:2181/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
      Enter username for jdbc:hive2://master:2181/default:
      Enter password for jdbc:hive2://master:2181/default:
      20/01/04 20:10:01 [main]: INFO jdbc.HiveConnection: Connected to master:10000
      Connected to: Apache Hive (version 3.1.0.3.1.4.0-315)
      Driver: Hive JDBC (version 3.1.0.3.1.4.0-315)
      Transaction isolation: TRANSACTION_REPEATABLE_READ
      Beeline version 3.1.0.3.1.4.0-315 by Apache Hive
      0: jdbc:hive2://master:2181/default> set fs.defaultFS;
      ----------------------------------

      set

      ----------------------------------

      fs.defaultFS=hdfs://master:8020

      ----------------------------------
      1 row selected (0.122 seconds)
      0: jdbc:hive2://master:2181/default> set fs.defaultFS=alluxio://master:19998;
      No rows affected (0.005 seconds)
      0: jdbc:hive2://master:2181/default> set fs.defaultFS;
      --------------------------------------

      set

      --------------------------------------

      fs.defaultFS=alluxio://master:19998

      --------------------------------------
      1 row selected (0.009 seconds)
      0: jdbc:hive2://master:2181/default> create table dual99(data string);
      INFO : Compiling command(queryId=hive_20200104201105_57ed827c-1e8f-4a39-a179-eda0f69e44d2): create table dual99(data string)
      INFO : Semantic Analysis Completed (retrial = false)
      INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
      INFO : Completed compiling command(queryId=hive_20200104201105_57ed827c-1e8f-4a39-a179-eda0f69e44d2); Time taken: 0.028 seconds
      INFO : Executing command(queryId=hive_20200104201105_57ed827c-1e8f-4a39-a179-eda0f69e44d2): create table dual99(data string)
      INFO : Starting task [Stage-0:DDL] in serial mode
      INFO : Completed executing command(queryId=hive_20200104201105_57ed827c-1e8f-4a39-a179-eda0f69e44d2); Time taken: 0.029 seconds
      INFO : OK
      No rows affected (0.095 seconds)
      0: jdbc:hive2://master:2181/default> show create table dual99;
      INFO : Compiling command(queryId=hive_20200104201114_db013c67-91ab-4465-9841-737722d53113): show create table dual99
      INFO : Semantic Analysis Completed (retrial = false)
      INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:createtab_stmt, type:string, comment:from deserializer)], properties:null)
      INFO : Completed compiling command(queryId=hive_20200104201114_db013c67-91ab-4465-9841-737722d53113); Time taken: 0.021 seconds
      INFO : Executing command(queryId=hive_20200104201114_db013c67-91ab-4465-9841-737722d53113): show create table dual99
      INFO : Starting task [Stage-0:DDL] in serial mode
      INFO : Completed executing command(queryId=hive_20200104201114_db013c67-91ab-4465-9841-737722d53113); Time taken: 0.013 seconds
      INFO : OK
      ----------------------------------------------------

      createtab_stmt

      ----------------------------------------------------

      CREATE TABLE `dual99`(
      `data` string)
      ROW FORMAT SERDE
      'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
      STORED AS INPUTFORMAT
      'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
      OUTPUTFORMAT
      'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
      LOCATION
      'hdfs://master:8020/warehouse/tablespace/managed/hive/dual99'
      TBLPROPERTIES (
      'bucketing_version'='2',
      'transient_lastDdlTime'='1578139865')

      ----------------------------------------------------
      13 rows selected (0.08 seconds)

      Attachments

        1. log.txt
          4 kB
          kairen lo

        Activity

          People

            kairen2 kairen lo
            kairen2 kairen lo
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: