Uploaded image for project: 'ActiveMQ Classic'
  1. ActiveMQ Classic
  2. AMQ-7010

No space left on device leading to broker shutdown even if ignored

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Won't Fix
    • 5.15.0, 5.15.1, 5.15.2, 5.15.3
    • None
    • KahaDB
    • None

    Description

      Recently, we upgraded our ActiveMQ instance from 5.13.2 to 5.15.3. We basically use an almost default broker setup with KahaDB. Due to our setup, we occasionally run into java.io.IOException: No space left on device, which until now was fine since the broker would properly ignore them using the DefaultIOExceptionHandler, and resume its work once space is available again.

      After upgrading to 5.15.3, however, we noticed that the broker would actually stop itself following a no space left exception. Digging a little bit, it seems that the change in behavior can be linked to a fix introduced in the 5.15.0 release, see AMQ-6625. With this fix, KahaDB would sometimes throw the following exception: java.io.IOException: Async Writer Thread Shutdown from its DataFileAppender, which, unfortunately, is not handled by the DefaultIOExceptionHandler unless ignoreAllErrors is set to true.

      This would basically mean that the DefaultIOExceptionHandler has become useless in catching only no space left exceptions, which I do not think is the intended behavior. 

      How to reproduce

      1. Start a broker with default config on a small partition
      2. Send a persistent message with web console to any queue
      3. Fill the partition by any mean
      4. Send more messages with web console, broker is eventually shut down

      See the attached broker logs.

      Attachments

        1. activemq_logs.txt
          31 kB
          Laurent Chiarello

        Issue Links

          Activity

            People

              Unassigned Unassigned
              lchdev Laurent Chiarello
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: