Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-21759

Confusing error message as a result of concurrent drop and create operations for managed tables

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • 2.3.1
    • None
    • Metastore
    • None

    Description

      It's possible for concurrent DROP TABLE and CREATE TABLE operations to yield the following sequence of events:

      1. The HMS receives a request to create managed table T
      2. The HMS will create a (HD)FS directory for T if one doesn't already exist
      3. The HMS will create a metadata entry for table T, making it visible to clients
      4. The client sends a DROP TABLE request for table T
      5. The table entry is dropped in the HMS
      6. The HMS will eventually drop the directory from (HD)FS

      If in between #5 and #6, the client sends another create table request for T, the request will begin to be processed; the table will not exist in the HMS (even though its directory exists), so the create will begin to proceed as if no underlying data exists. Eventually, #6 will complete, and if this coincides with the part of the second create table operation that enumerates the files in the directory (e.g. when generating the CreateTable notification event), the directory will not be there, potentially resulting in the following stack traces:

       

      21:44:29.974 [HMS - ERROR - pool-8-thread-7] (RetryingHMSHandler.java:200) MetaException(message:java.lang.RuntimeException: java.io.FileNotFoundException: File file:/tmp/dist-test-taskI7tPbq/test-tmp/master-stress-test.0.HmsSentryConfigurations_MasterStressTest.Test_2.1558388453623586-26465/minicluster-data/warehouse/table_c469fde3312d473fb7ac37f526e4dce0 does not exist)
      {{ at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:6189)}}
      {{ at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1553)}}
      {{ at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)}}
      {{ at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}}
      {{ at java.lang.reflect.Method.invoke(Method.java:498)}}
      {{ at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)}}
      {{ at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)}}
      {{ at com.sun.proxy.$Proxy24.create_table_with_environment_context(Unknown Source)}}
      {{ at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:11191)}}
      {{ at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:11175)}}
      {{ at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)}}
      {{ at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)}}
      {{ at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:626)}}
      {{ at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:621)}}
      {{ at java.security.AccessController.doPrivileged(Native Method)}}
      {{ at javax.security.auth.Subject.doAs(Subject.java:422)}}
      {{ at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)}}
      {{ at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:621)}}
      {{ at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)}}
      {{ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)}}
      {{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)}}
      {{ at java.lang.Thread.run(Thread.java:748)}}
      {{ Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: File file:/tmp/dist-test-taskI7tPbq/test-tmp/master-stress-test.0.HmsSentryConfigurations_MasterStressTest.Test_2.1558388453623586-26465/minicluster-data/warehouse/table_c469fde3312d473fb7ac37f526e4dce0 does not exist}}
      {{ at org.apache.hive.hcatalog.listener.DbNotificationListener$FileIterator.<init>(DbNotificationListener.java:222)}}
      {{ at org.apache.hive.hcatalog.listener.DbNotificationListener.onCreateTable(DbNotificationListener.java:156)}}
      {{ at org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier$13.notify(MetaStoreListenerNotifier.java:76)}}
      {{ at org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:175)}}
      {{ at org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:205)}}
      {{ at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1500)}}
      {{ at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1539)}}
      {{ ... 20 more}}
      {{ Caused by: java.io.FileNotFoundException: File file:/tmp/dist-test-taskI7tPbq/test-tmp/master-stress-test.0.HmsSentryConfigurations_MasterStressTest.Test_2.1558388453623586-26465/minicluster-data/warehouse/table_c469fde3312d473fb7ac37f526e4dce0 does not exist}}
      {{ at org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:455)}}
      {{ at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1537)}}
      {{ at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1580)}}
      {{ at org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:675)}}
      {{ at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1537)}}
      {{ at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1580)}}
      {{ at org.apache.hive.hcatalog.listener.DbNotificationListener$FileIterator.<init>(DbNotificationListener.java:219)}}
      {{ ... 26 more}}

       

      The client will see that the create table failed because of a MetaException, in which a file was not found. This is a somewhat confusing message, though the end behavior is cut and dry: there will be no HMS entry (should be rolled back as a part of the transaction) and no directory following this sequence of events.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              awong Andrew Wong
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated: