Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-12053

Expose event-processor error message in catalogd WebUI

    XMLWordPrintableJSON

Details

    Description

      Currently, we can only check logs to know why event-processor comes into ERROR state. E.g. here is the error message for IMPALA-12002:

      E0315 17:54:14.959136  2659 MetastoreEventsProcessor.java:865] Unexpected exception received while processing event
Java exception follows:
org.apache.impala.catalog.events.MetastoreNotificationException: EventId: 26044 EventType: COMMIT_COMPACTION_EVENT Unable to parse commit compaction message
        at org.apache.impala.catalog.events.MetastoreEvents$CommitCompactionEvent.<init>(MetastoreEvents.java:2684)
        at org.apache.impala.catalog.events.MetastoreEvents$MetastoreEventFactory.get(MetastoreEvents.java:223)
        at org.apache.impala.catalog.events.MetastoreEvents$MetastoreEventFactory.getFilteredEvents(MetastoreEvents.java:255)
        at org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:999)
        at org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:851)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.impala.catalog.DatabaseNotFoundException: Database 'test_acid_compaction_5c682e9f' not found
        at org.apache.impala.catalog.Catalog.getTable(Catalog.java:196)
        at org.apache.impala.catalog.events.MetastoreEvents$CommitCompactionEvent.<init>(MetastoreEvents.java:2679)
        ... 11 more
E0315 17:54:14.959192  2659 MetastoreEventsProcessor.java:1040] Notification event is null
W0315 17:54:15.959450  2659 MetastoreEventsProcessor.java:844] Event processing is skipped since status is ERROR. Last synced event id is 26042

      It'd be helpful to expose the stacktrace in catalogd's WebUI. Tests can also retrieve it to show more detailed failures instead of just complaining timeout.

      Attachments

        1. new_events_page.jpg
          275 kB
          Quanlong Huang
        2. catalog-events-webui.jpg
          304 kB
          Quanlong Huang

        Issue Links

          causes

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-12170 TestWebPage.test_webserver_hide_logs_link fails due to new /events link added in catalogd

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          relates to

          Sub-task - The sub-task of the issue IMPALA-7975 Improve supportability of the automatic invalidate feature

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              stigahuang Quanlong Huang
              stigahuang Quanlong Huang
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: