Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-527

Persistent Prov Repo should compress write-ahead-log files in chunks

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 0.1.0
    • Extensions
    • None

    Description

      Currently when we rollover a prov log, we compress the entire thing. This means that when we want to jump to a particular offset we have to open a GZIPInputStream and read through all of the data. If we instead compress the logs in chunks, we can actually jump to a particular chunk using FileInputStream.skip and then open a GZIPInputStream from there. Currently, this is by far the biggest bottleneck in the prov repo when doing queries.

      Attachments

        Activity

          People

            markap14 Mark Payne
            markap14 Mark Payne
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: