Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-8210

If Provenance Repository ages off a Lucene Index while it's being searched, can lead to index corruption

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.13.0
    • Core Framework
    • None

    Description

      When all events have been aged off from a Provenance Lucene Index, the lucene index can be deleted. However, before deleting it, the repository needs to ensure that there are no active readers or writers. Currently, the repository only ensures that there are no active writers. As a result, we can have an active Index Reader/Index Searcher and still delete the index directory. When this happens, we end up with an error similar to:

      org.apache.lucene.index.IndexNotFoundException: no segments* file found in MMapDirectory@/Users/mpayne/devel/nifi/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/target/6e442c6d-14f3-48bb-9500-534c136942aa lockFactory=org.apache.lucene.store.NativeFSLockFactory@161b062a: files: []
      	at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:675)
      	at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:81)
      	at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:63)
      	at org.apache.nifi.provenance.lucene.StandardIndexManager.borrowIndexSearcher(StandardIndexManager.java:128)
      ...

      We need to ensure that we account for any Readers/Searchers that are active also

      Attachments

        Issue Links

          Activity

            People

              markap14 Mark Payne
              markap14 Mark Payne
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 40m
                  40m