Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-7407

Explore switching doc values to an iterator API

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 7.0
    • None
    • New

    Description

      I think it could be compelling if we restricted doc values to use an
      iterator API at read time, instead of the more general random access
      API we have today:

      • It would make doc values disk usage more of a "you pay for what
        what you actually use", like postings, which is a compelling
        reduction for sparse usage.
      • I think codecs could compress better and maybe speed up decoding
        of doc values, even in the non-sparse case, since the read-time
        API is more restrictive "forward only" instead of random access.
      • We could remove getDocsWithField entirely, since that's
        implicit in the iteration, and the awkward "return 0 if the
        document didn't have this field" would go away.
      • We can remove the annoying thread locals we must make today in
        CodecReader, and close the trappy "I accidentally shared a
        single XXXDocValues instance across threads", since an iterator is
        inherently "use once".
      • We could maybe leverage the numerous optimizations we've done for
        postings over time, since the two problems ("iterate over doc ids
        and store something interesting for each") are very similar.

      This idea has come up many in the past, e.g. LUCENE-7253 is a recent
      example, and very early iterations of doc values started with exactly
      this

      However, it's a truly enormous change, likely 7.0 only. Or maybe we
      could have the new iterator APIs also ported to 6.x side by side with
      the deprecate existing random-access APIs.

      Attachments

        1. LUCENE-7407.patch
          1.20 MB
          Michael McCandless

        Issue Links

          breaks

          Bug - A problem which impairs or prevents the functions of the product. SOLR-9599 DocValues performance regression with new iterator API

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. SOLR-10596 unlque and hll functions don't work after first bucket

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SOLR-11664 range facets with with sub aggregations on string fields give incorrect results

          • Blocker - Blocks development and/or testing work, production could not run
          • Closed
          is related to

          Sub-task - The sub-task of the issue SOLR-9837 Performance regression of numeric field uninversion time

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SOLR-9582 TestSortingResponseWriter.testSortingOutput() failure: docs were sent out-of-order

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. LUCENE-7835 ToChildBlockJoinSortField to sort children by a parent field

          • Major - Major loss of function.
          • Patch Available

          Improvement - An improvement or enhancement to an existing feature or task. SOLR-13024 ValueSourceAugmenter - avoid creating new FunctionValues per doc

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. LUCENE-7253 Make sparse doc values and segments merging more efficient

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. LUCENE-7474 Improve doc values writers

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. LUCENE-10534 MinFloatFunction / MaxFloatFunction calls exists twice

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. LUCENE-10542 FieldSource exists implementations can avoid value retrieval

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. SOLR-9628 Trie fields have unset lastDocId

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. LUCENE-7871 false positive match BlockJoinSelector[SortedDV] when child value is absent

          • Major - Major loss of function.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. LUCENE-7835 ToChildBlockJoinSortField to sort children by a parent field

          • Major - Major loss of function.
          • Patch Available

          Improvement - An improvement or enhancement to an existing feature or task. LUCENE-7457 Default doc values format should optimize for iterator access

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. LUCENE-5542 Explore making DVConsumer sparse-aware

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. LUCENE-7459 LegacyNumericDocValuesWrapper should only check bits when the value is != 0

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. LUCENE-7461 Refactor doc values queries to better use the new doc values APIs

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. LUCENE-7462 Faster search APIs for doc values

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. LUCENE-7463 Create a Lucene70DocValuesFormat

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. LUCENE-7475 Sparse norms

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. LUCENE-7489 Improve sparsity support of Lucene70DocValuesFormat

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Wish - General wishlist item. LUCENE-7460 Should SortedNumericDocValues expose a per-document random-access API?

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            People

              mikemccand Michael McCandless
              mikemccand Michael McCandless
              Votes:
              1 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: