Uploaded image for project: 'Apache IoTDB'
  1. Apache IoTDB
  2. IOTDB-106

How about add a new strategy to close data files in IoTDB according to the memory usage

    XMLWordPrintableJSON

Details

    • Wish
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None

    Description

      Different with other LSM-based systems, IoTDB will flush multiple memtable into one TsFile file. 

      The TsFile can support append memtables one by one unless a "close file" operation is called.

      Suppose $n$ memtables  are flushed into one un-closed TsFile (called unsealed file in the source code), these memtables' metadata will be hold in memory, which may lead to high cost of memory under some heavy workloads.

       

      Now there are 3 strategies to trigger a "close" operation: (1) call the operation per `period_time_for_flush_in_second` seconds (configured in iotdb-engine.properties); (2) if the unsealed tsfile size reaches to `bufferwrite_file_size_threshold` (also configured in the same setting file); (3) user call `flush` in the iotdb-cli shell.

       

      How about add a new strategy:  only allow $max$ metadata in memory. If the number in memory exceeds $max$, seal the file. In advance, $max$ can be calculated according to the available memory. 

       

       

      Attachments

        Activity

          People

            Unassigned Unassigned
            jixuan1989 xiangdong Huang
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: