Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
Description
In hbase- default.xml
<property> <name>hbase.regionserver.logroll.period</name> <value>3600000</value> <description>Period at which we will roll the commit log regardless of how many edits it has.</description> </property>
Taking hbase.regionserver.logroll.period set to -1 in hbase-site.xml as an example, it is normal to start hbase.
However, there are a large number of repeated wal rolling operation records in the log file.
2022-08-06 15:39:37,172 INFO [regionserver/xmw1-kb310-65:16020.logRoller] wal.AbstractFSWAL: Rolled WAL /home/xmw/hbase-2.2.2/hbase-tmp/WALs/xmw1-kb310-65,16020,1659771570212/xmw1-kb310-65%2C16020%2C1659771570212.1659771577156 with entries=0, filesize=83 B; new WAL /home/xmw/hbase-2.2.2/hbase-tmp/WALs/xmw1-kb310-65,16020,1659771570212/xmw1-kb310-65%2C16020%2C1659771570212.1659771577167 2022-08-06 15:39:37,172 INFO [regionserver/xmw1-kb310-65:16020.logRoller] wal.AbstractFSWAL: Archiving file:/home/xmw/hbase-2.2.2/hbase-tmp/WALs/xmw1-kb310-65,16020,1659771570212/xmw1-kb310-65%2C16020%2C1659771570212.1659771577156 to file:/home/xmw/hbase-2.2.2/hbase-tmp/oldWALs/xmw1-kb310-65%2C16020%2C1659771570212.1659771577156 2022-08-06 15:39:37,177 INFO [regionserver/xmw1-kb310-65:16020.logRoller] wal.AbstractFSWAL: Rolled WAL /home/xmw/hbase-2.2.2/hbase-tmp/WALs/xmw1-kb310-65,16020,1659771570212/xmw1-kb310-65%2C16020%2C1659771570212.meta.1659771577160.meta with entries=0, filesize=83 B; new WAL /home/xmw/hbase-2.2.2/hbase-tmp/WALs/xmw1-kb310-65,16020,1659771570212/xmw1-kb310-65%2C16020%2C1659771570212.meta.1659771577172.meta 2022-08-06 15:39:37,177 INFO [regionserver/xmw1-kb310-65:16020.logRoller] wal.AbstractFSWAL: Archiving file:/home/xmw/hbase-2.2.2/hbase-tmp/WALs/xmw1-kb310-65,16020,1659771570212/xmw1-kb310-65%2C16020%2C1659771570212.meta.1659771577160.meta to file:/home/xmw/hbase-2.2.2/hbase-tmp/oldWALs/xmw1-kb310-65%2C16020%2C1659771570212.meta.1659771577160.meta 2022-08-06 15:39:37,182 INFO [regionserver/xmw1-kb310-65:16020.logRoller] wal.AbstractFSWAL: Rolled WAL /home/xmw/hbase-2.2.2/hbase-tmp/WALs/xmw1-kb310-65,16020,1659771570212/xmw1-kb310-65%2C16020%2C1659771570212.1659771577167 with entries=0, filesize=83 B; new WAL /home/xmw/hbase-2.2.2/hbase-tmp/WALs/xmw1-kb310-65,16020,1659771570212/xmw1-kb310-65%2C16020%2C1659771570212.1659771577177 2022-08-06 15:39:37,182 INFO [regionserver/xmw1-kb310-65:16020.logRoller] wal.AbstractFSWAL: Archiving file:/home/xmw/hbase-2.2.2/hbase-tmp/WALs/xmw1-kb310-65,16020,1659771570212/xmw1-kb310-65%2C16020%2C1659771570212.1659771577167 to file:/home/xmw/hbase-2.2.2/hbase-tmp/oldWALs/xmw1-kb310-65%2C16020%2C1659771570212.1659771577167 2022-08-06 15:39:37,188 INFO [regionserver/xmw1-kb310-65:16020.logRoller] wal.AbstractFSWAL: Rolled WAL /home/xmw/hbase-2.2.2/hbase-tmp/WALs/xmw1-kb310-65,16020,1659771570212/xmw1-kb310-65%2C16020%2C1659771570212.meta.1659771577172.meta with entries=0, filesize=83 B; new WAL /home/xmw/hbase-2.2.2/hbase-tmp/WALs/xmw1-kb310-65,16020,1659771570212/xmw1-kb310-65%2C16020%2C1659771570212.meta.1659771577182.meta 2022-08-06 15:39:37,188 INFO [regionserver/xmw1-kb310-65:16020.logRoller] wal.AbstractFSWAL: Archiving file:/home/xmw/hbase-2.2.2/hbase-tmp/WALs/xmw1-kb310-65,16020,1659771570212/xmw1-kb310-65%2C16020%2C1659771570212.meta.1659771577172.meta to file:/home/xmw/hbase-2.2.2/hbase-tmp/oldWALs/xmw1-kb310-65%2C16020%2C1659771570212.meta.1659771577172.meta 2022-08-06 15:39:37,196 INFO [regionserver/xmw1-kb310-65:16020.logRoller] wal.AbstractFSWAL: Rolled WAL /home/xmw/hbase-2.2.2/hbase-tmp/WALs/xmw1-kb310-65,16020,1659771570212/xmw1-kb310-65%2C16020%2C1659771570212.1659771577177 with entries=0, filesize=83 B; new WAL /home/xmw/hbase-2.2.2/hbase-tmp/WALs/xmw1-kb310-65,16020,1659771570212/xmw1-kb310-65%2C16020%2C1659771570212.1659771577188 2022-08-06 15:39:37,197 INFO [regionserver/xmw1-kb310-65:16020.logRoller] wal.AbstractFSWAL: Archiving file:/home/xmw/hbase-2.2.2/hbase-tmp/WALs/xmw1-kb310-65,16020,1659771570212/xmw1-kb310-65%2C16020%2C1659771570212.1659771577177 to file:/home/xmw/hbase-2.2.2/hbase-tmp/oldWALs/xmw1-kb310-65%2C16020%2C1659771570212.1659771577177 2022-08-06 15:39:37,201 INFO [regionserver/xmw1-kb310-65:16020.logRoller] wal.AbstractFSWAL: Rolled WAL /home/xmw/hbase-2.2.2/hbase-tmp/WALs/xmw1-kb310-65,16020,1659771570212/xmw1-kb310-65%2C16020%2C1659771570212.meta.1659771577182.meta with entries=0, filesize=83 B; new WAL /home/xmw/hbase-2.2.2/hbase-tmp/WALs/xmw1-kb310-65,16020,1659771570212/xmw1-kb310-65%2C16020%2C1659771570212.meta.1659771577197.meta 2022-08-06 15:39:37,201 INFO [regionserver/xmw1-kb310-65:16020.logRoller] wal.AbstractFSWAL: Archiving file:/home/xmw/hbase-2.2.2/hbase-tmp/WALs/xmw1-kb310-65,16020,1659771570212/xmw1-kb310-65%2C16020%2C1659771570212.meta.1659771577182.meta to file:/home/xmw/hbase-2.2.2/hbase-tmp/oldWALs/xmw1-kb310-65%2C16020%2C1659771570212.meta.1659771577182.meta
When using the service, it works fine and close normally.
According to the above phenomenon, we believe that the phenomenon is related to the WAL rolling of regionserver. After code review, we found that in the constructor of hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java. After reading the configuration item hbase.regionserver.logroll.period, it is not checked whether the value is normal. In the subsequent run() function, the current time is directly compared to the last rollback time with hbase.regionserver.logroll.period to determine whether to perform the WAL rollback operation, so that no matter what the current time is, The rollback operation will be performed, resulting in a large number of records in the LOG, and it is very likely to occupy too many resources. And no prompt message appears to remind us of the problem.
Therefore, we propose to add the judgment of hbase.regionserver.logroll.period on the constructor of LogRoller. If the parameter LogRoller.rollPeriod affected by this configuration item is less than zero, an error message will be output and he relevant exception can be thrown.
This way, if the hbase.regiononserver.logroll.period is less than 0, an exception can be thrown and an error message logged, stopping the log from having a large number of duplicate wal roll operation records.