Unbounded SDF wrapper causes performance regression on DirectRunner

There is a significant performance regression when switching from UnboundedSource to Unbounded SDF wrapper. So far there are 2 IOs reported:

• Pubsub Read: https://lists.apache.org/thread.html/re6b0941a8b4951293a0327ce9b25e607cafd6e45b69783f65290edee%40%3Cdev.beam.apache.org%3E
• Kafka Read: https://the-asf.slack.com/archives/C9H0YNP3P/p1606155042346600