Spark creates lot of tiny blocks for a single driverLog file of size less than a dfs.blocksize

Upon reviewing the spark code, found that /user/spark/driverLogs are synced to HDFS with hsync option as shown below.

 hdfsStream.hsync(EnumSet.allOf(classOf[HdfsDataOutputStream.SyncFlag]))

Ref: https://github.com/apache/spark/blob/a3c04ec1145662e4227d57cd953bffce96b8aad7/core/src/main/scala/org/apache/spark/util/logging/DriverLogger.scala

As a result of this we see lot of tiny blocks getting synced every 5 seconds with a new block. So we see a small HDFS file with 8 blocks as shown in the below example.

[root@ccycloud-3.smajeti.root.comops.site subdir0]# hdfs fsck /user/spark/driverLogs/application_1710495774861_0002_driver.log
Connecting to namenode via https://ccycloud-3.smajeti.root.comops.site:20102/fsck?ugi=hdfs&path=%2Fuser%2Fspark%2FdriverLogs%2Fapplication_1710495774861_0002_driver.log
FSCK started by hdfs (auth:KERBEROS_SSL) from /10.140.136.139 for path /user/spark/driverLogs/application_1710495774861_0002_driver.log at Thu Mar 28 06:37:29 UTC 2024

Status: HEALTHY
 Number of data-nodes:	4
 Number of racks:		1
 Total dirs:			0
 Total symlinks:		0

Replicated Blocks:
 Total size:	157574 B
 Total files:	1
 Total blocks (validated):	8 (avg. block size 19696 B)
 Minimally replicated blocks:	8 (100.0 %) 

HdfsDataOutputStream.SyncFlag includes two flags UPDATE_LENGTH and END_BLOCK. This has been an expected behavior for some time now and these flags will help visualize the latest size of the HDFS Driver log file and to achieve that, blocks are being ended/closed every 5-second sync. Every new sync will create a new block for the same HDFS driver log file. This hysnc behavior was started after fixing SPARK-29105 (SHS may delete driver log file of in-progress application) 5 years back.

But this leaves Namenode to manage a lot of meta and becomes an overhead at times in large clusters.

public static enum SyncFlag {
        UPDATE_LENGTH,
        END_BLOCK;

        private SyncFlag() {
        }
    }

I don't see any configurable option to avoid this and avoiding this type of hsync may have some side effects in spark as we saw SPARK-29105 bug.
We only have two options that needs manual intevention
1. Keep cleaning these Driver logs after some time
2. Keep merging these small block files into files with 128MB 

Can we provide some customizable option to merge these blocks while closing the spark-shell or during closing the driver log file?