Multiple time sources with fallback behavior between them

Todd Lipcon suggested an alternative approach to configure and select HybridClock's time source.

Kudu servers could maintain multiple time sources and switch between them with a fallback behavior. The default or preferred time source might be any of the existing ones (e.g., the built-in client), but when it's not available, another available time source is selected (e.g., system – the NTP-synchronized local clock). Switching between time sources can be done:

• only upon startup/initialization
• upon startup/initialization and later during normal run time

The advantages are:

• easier deployment and configuration of Kudu clusters
• simplified upgrade path from older releases using system time source to newer releases using builtin time source by default

There are downsides, though. Since the new way of maintaining time source is more dynamic, it can:

• mask various configuration or network issues
• result in different time source within the same Kudu cluster due to transient issues
• introduce extra startup delay