DataNode supports more fine-grained dataset lock based on blockid

Recently, we used NvmeSSD as volumes in datanodes and performed some stress tests.

We found that NvmeSSD and HDD disks achieve similar performance when create lots of small files, such as 10KB.

This phenomenon is counterintuitive.  After analyzing the metric monitoring , we found that fsdataset lock became the bottleneck in high concurrency scenario.

Currently, we have two level locks which are BLOCK_POOL and VOLUME.  We can further split the volume lock to DIR lock.

DIR lock is defined as below： given a blockid, we can determine which subdir this block will be placed in finalized dir. We just use subdir[0-31]/subdir[0-31] as the

name of DIR lock.

More details, please refer to method DatanodeUtil#idToBlockDir：

  public static File idToBlockDir(File root, long blockId) {
    int d1 = (int) ((blockId >> 16) & 0x1F);
    int d2 = (int) ((blockId >> 8) & 0x1F);
    String path = DataStorage.BLOCK_SUBDIR_PREFIX + d1 + SEP +
        DataStorage.BLOCK_SUBDIR_PREFIX + d2;
    return new File(root, path);
  } 

The performance comparison is as below:

experimental setup:

3 DataNodes with single disk.

10 Cients concurrent write and delete files after writing.

550 threads per Client.