Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-6078

IMPALA query fails if a scratch directory goes bad with a disk failure

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • Impala 2.9.0
    • Impala 2.9.0
    • Backend
    • None
    • ghx-label-3

    Description

      Impala was configured with two scratch directories on 2 different disks.

      We simulated disk failure on one of the Scratch directories and it became inaccessible

      In such scenarios, We get the following error when a query tries to spill to the scratch directories and the query fails:

      >>>>>>>>>
      > select avg(t1.a1), max(t2.a2) from impala_disk_test.impala_tst1 as t1 full join impala_disk_test.impala_tst1 as t2 ON t1.a1=t2.a1;
      Query: select avg(t1.a1), max(t2.a2) from impala_disk_test.impala_tst1 as t1 full join impala_disk_test.impala_tst1 as t2 ON t1.a1=t2.a1
      Cancelling Query
      Application Exception : Default TException.
      t2.a1;t.impala_tst1 as t1 full join impala_disk_test.impala_tst1 as t2 ON t1.a1=t
      Query: select avg(t1.a1), max(t2.a2) from impala_disk_test.impala_tst1 as t1 full join impala_disk_test.impala_tst1 as t2 ON t1.a1=t2.a1
      WARNINGS:
      No usable temporary files: space could not be allocated on any temporary device.
      Create file /srv/impala2/impala/impala-scratch/5b431fe3571b9bf1:ef5b367fb74fd1b9_597f261f-792e-4ade-8b32-f4198c945142 failed with errno=5 description=Error(5): Input/output error

      No usable temporary files: space could not be allocated on any temporary device.
      >>>>>>>>>

      It does not look good from HA point of view to have an Impala daemon in the cluster that will sometimes fail queries just because it has a disk failed a few hour ago.

      If a disk fails during a query there is really nothing you can do, This request is to address query failures that started after the disk has failed.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              kamal.chandrasekaran kamal Chandrasekaran
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: