Milestone 3: Reading “original files”

“Original files” don’t store special ACID columns, that means we need to auto-generate those values. Actually we only need to auto-generate the record id: (originalTransaction, bucket, rowId).

• originalTransaction: can be parsed from the containing directory
  • If it’s the table root directory then originalTransaction is 0

• Bucket: it’s the bit-packed value of (bucket codec version, bucket id, and statement id)
  • Bucket codec version is 1
  • Bucket id can be parsed from the filename
  • Statement id can be parsed from the delta directory:
    • delta_<min_writeid><max_writeid><statement_id>
    • (min_writeid = max_writeid for original files)

• rowId: zero-based for each bucket, if there are multiple files in a single bucket:
  • List all the files belonging to the bucket
  • First file’s first row id is 0
  • Next file’s first row id is the row count of the first file
  • And so on

The frontend should generate the base record ID for each file and propagate that information to the scanners. Therefore the scanners would know if they are scanning files in full ACID format or raw format. The ORC scanner needs to be changed in order to generate and fill the ACID columns for original files.