Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-6517 Snapshot support for Ozone
  3. HDDS-8786

Encounter error of OzoneManagerDoubleBuffer Terminating during snapshot creation

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Cannot Reproduce
    • None
    • None
    • Ozone Manager

    Description

      When creating snapshot, encountered following error in OM:

      2023-06-08 02:01:14,771 [OMDoubleBufferFlushThread] ERROR org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer: Terminating with exit statu
      s 1: During flush to DB encountered error in OMDoubleBuffer flush thread OMDoubleBufferFlushThread when handling OMRequest: cmdType: CreateSnaps
      hot
      traceID: ""
      success: true
      status: OK
      CreateSnapshotResponse {
        snapshotInfo {
          snapshotID: "3ce30204-da71-41ad-9e78-607c688e8c5c"
          name: "snap5670"
          volumeName: "vol2"
          bucketName: "bucket2"
          snapshotStatus: SNAPSHOT_ACTIVE
          creationTime: 1686189673557
          deletionTime: 18446744073709551615
          pathPreviousSnapshotID: "180bd9f4-a83e-4bfb-97b3-a19d0eef0450"
          globalPreviousSnapshotID: "180bd9f4-a83e-4bfb-97b3-a19d0eef0450"
          snapshotPath: "vol2/bucket2"
          checkpointDir: "-3ce30204-da71-41ad-9e78-607c688e8c5c"
          dbTxSequenceNumber: 3269138
        }
      }
      
      java.io.IOException: Rocks Database is closed
              at org.apache.hadoop.hdds.utils.db.RocksDatabase.assertClose(RocksDatabase.java:442)
              at org.apache.hadoop.hdds.utils.db.RocksDatabase.newIterator(RocksDatabase.java:827)
              at org.apache.hadoop.hdds.utils.db.RDBTable.iterator(RDBTable.java:194)
              at org.apache.hadoop.hdds.utils.db.TypedTable.iterator(TypedTable.java:361)
              at org.apache.hadoop.ozone.om.OmSnapshotManager.deleteKeysFromDelKeyTableInSnapshotScope(OmSnapshotManager.java:573)
              at org.apache.hadoop.ozone.om.OmSnapshotManager.createOmSnapshotCheckpoint(OmSnapshotManager.java:394)
              at org.apache.hadoop.ozone.om.response.snapshot.OMSnapshotCreateResponse.addToDBBatch(OMSnapshotCreateResponse.java:81)
              at org.apache.hadoop.ozone.om.response.OMClientResponse.checkAndUpdateDB(OMClientResponse.java:73)
              at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.lambda$5(OzoneManagerDoubleBuffer.java:414)
      at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.addToBatchWithTrace(OzoneManagerDoubleBuffer.java:237)
              at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.addToBatch(OzoneManagerDoubleBuffer.java:412)
              at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushBatch(OzoneManagerDoubleBuffer.java:333)
              at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushCurrentBuffer(OzoneManagerDoubleBuffer.java:312)
              at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushTransactions(OzoneManagerDoubleBuffer.java:277)
              at java.lang.Thread.run(Thread.java:748) 

      Attachments

        Activity

          People

            hemantk Hemant Kumar
            dteng Dave Teng
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: