Flaky test: TestFanOutOneBlockAsyncDFSOutput (Part 2)

The following test sometimes fails for me when running locally with Maven:

TestFanOutOneBlockAsyncDFSOutput.testRecover()

I can't really figure out the reason, but it's probably a side effect of the preceding test: testConnectToDatanodeFailed(). This test also restarts one of the datanodes in the MiniDFS cluster just like testRecover() and it somehow causes the failure.

java.lang.AssertionError: flush should fail
	at org.junit.Assert.fail(Assert.java:89)
	at org.apache.hadoop.hbase.io.asyncfs.TestFanOutOneBlockAsyncDFSOutput.testRecover(TestFanOutOneBlockAsyncDFSOutput.java:154)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

The flush() call is testRecover() should fail, because we restart one of the DN in the DFS cluster which is expected to brake connection. It succeeds though if the preceding test already restarted a DN. No matter which DN we restart, even if they're different, the error occurs.

I also tried to add CLUSTER.waitDatanodeFullyStarted() at the end of testConnectToDatanodeFailed(), looks like it made the tests slightly more stable, but didn't help fully.