Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-1940

Closing open container via scmcli gives false error message

    XMLWordPrintableJSON

Details

    Description

      scmcli close prints an error message about invalid state transition after it had successfully closed the container.

      CLI
      $ ozone scmcli info 2
      ...
      Container State: OPEN
      ...
      
      $ ozone scmcli close 2
      ...
      client-09830A377AA9->f27bf787-8711-41d4-b0fd-3ef50b5c076f: receive RaftClientReply:client-09830A377AA9->f27bf787-8711-41d4-b0fd-3ef50b5c076f@group-7831D6F2EF1B, cid=0, SUCCESS, logIndex=11, commits[f27bf787-8711-41d4-b0fd-3ef50b5c076f:c12, 37ba33fe-c9ed-4ac2-a6e5-57ce658168b4:c11, feb68ba4-0a8a-4eda-9915-7dc090e5f46c:c11]
      Failed to update container state #2, reason: invalid state transition from state: CLOSED upon event: CLOSE.
      
      $ ozone scmcli info 2
      ...
      Container State: CLOSED
      ...
      
      logs
      scm_1  | 2019-08-09 15:15:01 [IPC Server handler 1 on 9860] INFO  SCMClientProtocolServer:366 - Object type container id 1 op close new stage begin
      dn3_1  | 2019-08-09 15:15:02 [RatisApplyTransactionExecutor 1] INFO  Container:356 - Container 1 is closed with bcsId 3.
      dn1_1  | 2019-08-09 15:15:02 [RatisApplyTransactionExecutor 1] INFO  Container:356 - Container 1 is closed with bcsId 3.
      scm_1  | 2019-08-09 15:15:02 [EventQueue-IncrementalContainerReportForIncrementalContainerReportHandler] INFO  IncrementalContainerReportHandler:176 - Moving container #1 to CLOSED state, datanode feb68ba4-0a8a-4eda-9915-7dc090e5f46c{ip: 10.5.1.6, host: ozone-static_dn3_1.ozone-static_net, networkLocation: /default-rack, certSerialId: null} reported CLOSED replica.
      dn2_1  | 2019-08-09 15:15:02 [RatisApplyTransactionExecutor 1] INFO  Container:356 - Container 1 is closed with bcsId 3.
      scm_1  | 2019-08-09 15:15:02 [IPC Server handler 3 on 9860] INFO  SCMClientProtocolServer:366 - Object type container id 1 op close new stage complete
      scm_1  | 2019-08-09 15:15:02 [IPC Server handler 3 on 9860] ERROR ContainerStateManager:335 - Failed to update container state #1, reason: invalid state transition from state: CLOSED upon event: CLOSE.
      scm_1  | 2019-08-09 15:15:02 [IPC Server handler 3 on 9860] INFO  Server:2726 - IPC Server handler 3 on 9860, call Call#3 Retry#0 org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocol.notifyObjectStageChange from 10.5.0.71:57746
      scm_1  | org.apache.hadoop.hdds.scm.exceptions.SCMException: Failed to update container state #1, reason: invalid state transition from state: CLOSED upon event: CLOSE.
      scm_1  | 	at org.apache.hadoop.hdds.scm.container.ContainerStateManager.updateContainerState(ContainerStateManager.java:336)
      scm_1  | 	at org.apache.hadoop.hdds.scm.container.SCMContainerManager.updateContainerState(SCMContainerManager.java:312)
      scm_1  | 	at org.apache.hadoop.hdds.scm.server.SCMClientProtocolServer.notifyObjectStageChange(SCMClientProtocolServer.java:379)
      scm_1  | 	at org.apache.hadoop.ozone.protocolPB.StorageContainerLocationProtocolServerSideTranslatorPB.notifyObjectStageChange(StorageContainerLocationProtocolServerSideTranslatorPB.java:219)
      scm_1  | 	at org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos$StorageContainerLocationProtocolService$2.callBlockingMethod(StorageContainerLocationProtocolProtos.java:16398)
      scm_1  | 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
      scm_1  | 	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
      scm_1  | 	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
      scm_1  | 	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
      scm_1  | 	at java.base/java.security.AccessController.doPrivileged(Native Method)
      scm_1  | 	at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
      scm_1  | 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
      scm_1  | 	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
      

      Attachments

        Issue Links

          Activity

            People

              nanda Nandakumar
              adoroszlai Attila Doroszlai
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m