[Flakey Tests] Add test to check we work properly when port clash setting up thriftserver

My hack to add retry when thrift setup has a port clash over in HBASE-24079 didn't work. Add a test and fix what it turns up.