Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-7733

TestInsertParquetQueries.test_insert_parquet is flaky in S3 due to rename

    XMLWordPrintableJSON

Details

    Description

      I see two examples in the past two months or so where this test fails due to a rename error on S3. The test's stacktrace looks like this:

      query_test/test_insert_parquet.py:112: in test_insert_parquet
          self.run_test_case('insert_parquet', vector, unique_database, multiple_impalad=True)
      common/impala_test_suite.py:408: in run_test_case
          result = self.__execute_query(target_impalad_client, query, user=user)
      common/impala_test_suite.py:625: in __execute_query
          return impalad_client.execute(query, user=user)
      common/impala_connection.py:160: in execute
          return self.__beeswax_client.execute(sql_stmt, user=user)
      beeswax/impala_beeswax.py:176: in execute
          handle = self.__execute_query(query_string.strip(), user=user)
      beeswax/impala_beeswax.py:350: in __execute_query
          self.wait_for_finished(handle)
      beeswax/impala_beeswax.py:371: in wait_for_finished
          raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
      E   ImpalaBeeswaxException: ImpalaBeeswaxException:
      E    Query aborted:Error(s) moving partition files. First error (of 1) was: Hdfs op (RENAME s3a://<removed>/test_insert_parquet_968f37fe.db/orders_insert_table/_impala_insert_staging/4e45cd68bcddd451_3c7156ed00000000/.4e45cd68bcddd451-3c7156ed00000002_803672621_dir/4e45cd68bcddd451-3c7156ed00000002_448261088_data.0.parq TO s3a://<removed>/test-warehouse/test_insert_parquet_968f37fe.db/orders_insert_table/4e45cd68bcddd451-3c7156ed00000002_448261088_data.0.parq) failed, error was: s3a://<removed>/test-warehouse/test_insert_parquet_968f37fe.db/orders_insert_table/_impala_insert_staging/4e45cd68bcddd451_3c7156ed00000000/.4e45cd68bcddd451-3c7156ed00000002_803672621_dir/4e45cd68bcddd451-3c7156ed00000002_448261088_data.0.parq
      E   Error(5): Input/output error

      Since we know this happens once in a while, some ideas to deflake it:

      • retry
      • check for this specific issue... if we think its platform flakiness, then we should skip it.

      Attachments

        Issue Links

          Activity

            People

              lenisha Lenisha Gandhi
              vukercegovac Vuk Ercegovac
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: