Uploaded image for project: 'Apache IoTDB'
  1. Apache IoTDB
  2. IOTDB-1003

Split WAL buffer to better absorb ingestion

    XMLWordPrintableJSON

Details

    Description

      In the current implementation, we provide only one WAL buffer for each MemTable, which means if the buffer is full, we will have to wait until the buffer is flushed before we can write the next log, thus creating spikes in ingestions.

      So, it is benefitial to split the WAL buffer into a working one and a flusing one, and if last flush has completed before we ran out of the current working one, we can directly swap them, and continue writes.

      There is no point to split the buffer into more than 2 buffers, as the total capacity of the WALs are strictly controlled to bound data loss.

      Attachments

        Issue Links

          Activity

            People

              jt2594838 Tian Jiang
              jt2594838 Tian Jiang
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: