[HDFS-17003] Erasure Coding: invalidate wrong block after reporting bad blocks from datanode - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Critical
Resolution: Fixed
Affects Version/s: 3.4.0, 3.3.6
Fix Version/s: 3.4.0, 3.3.6
Component/s: namenode
Labels:
- pull-request-available

Target Version/s:

3.3.6
Hadoop Flags:

Reviewed

Description

After receiving reportBadBlocks RPC from datanode, NameNode compute wrong block to invalidate. It is a dangerous behaviour and may cause data loss. Some logs in our production as below:

NameNode log:

2023-05-08 21:23:49,112 INFO org.apache.hadoop.hdfs.StateChange: *DIR* reportBadBlocks for block: BP-932824627-xxxx-1680179358678:blk_-9223372036848404320_1471186 on datanode: datanode1:50010

2023-05-08 21:23:49,183 INFO org.apache.hadoop.hdfs.StateChange: *DIR* reportBadBlocks for block: BP-932824627-xxxx-1680179358678:blk_-9223372036848404319_1471186 on datanode: datanode2:50010

datanode1 log:

2023-05-08 21:23:49,088 WARN org.apache.hadoop.hdfs.server.datanode.VolumeScanner: Reporting bad BP-932824627-xxxx-1680179358678:blk_-9223372036848404320_1471186 on /data7/hadoop/hdfs/datanode

2023-05-08 21:24:00,509 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Failed to delete replica blk_-9223372036848404319_1471186: ReplicaInfo not found.

This phenomenon can be reproduced.

Attachments

Issue Links

links to

GitHub Pull Request #5643

Activity

People

Assignee:: farmmamba

Reporter:: farmmamba

Votes:: 0 Vote for this issue

Watchers:: 7 Start watching this issue

Dates

Created:: 08/May/23 12:42

Updated:: 28/Jan/24 02:19

Resolved:: 08/Jun/23 10:08