Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
When the LockProcedure was scheduled. It will called TimeoutExecutorThread#add method.
public void add(Procedure<TEnvironment> procedure) { assert procedure.getState() == ProcedureState.WAITING_TIMEOUT; LOG.info("ADDED {}; timeout={}, timestamp={}", procedure, procedure.getTimeout(), procedure.getTimeoutTimestamp()); queue.add(new DelayedProcedure<>(procedure)); }
But LockProcedure's unlock method may called before it was scheduled. And it will change the state to RUNNABLE. This will make the assert error.
// Can be called before procedure gets scheduled, in which case, the execute() will finish // immediately and release the underlying locks. public void unlock(final MasterProcedureEnv env) { unlock.set(true); locked.set(false); // Maybe timeout already awakened the event and the procedure has finished. synchronized (event) { if (!event.isReady()) { LOG.info("Update procedure {} state to RUNNABLE", this); setState(ProcedureProtos.ProcedureState.RUNNABLE); event.wake(env.getProcedureScheduler()); } } }
I added log in these methods.
2019-05-28 15:34:16,179 INFO [Time-limited test] locking.LockProcedure(223): Update procedure pid=21, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=namespace:table, type=EXCLUSIVE state to RUNNABLE
2019-05-28 15:34:16,179 DEBUG [Time-limited test] procedure.MasterProcedureScheduler(352): Add TableQueue(namespace:table, xlock=true (21) sharedLock=0 size=1) to run queue because: pid=21, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=namespace:table, type=EXCLUSIVE has lock
2019-05-28 15:34:16,179 INFO [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1967): ASSERT pid=21
java.lang.AssertionError
at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.add(TimeoutExecutorThread.java:82)
at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1690)
at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1395)
at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:78)
at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1965)
Attachments
Issue Links
- links to