Running a Large Job with Speculation On Causes Executor Heartbeats to Time Out on Driver

Running a large Spark job with speculation turned on was causing executor heartbeats to time out on the driver end after sometime and eventually, after hitting the max number of executor failures, the job would fail.