Support service mesh istio with Flink kubernetes (both native and operator) for secure communications

Currently Flink Native Kubernetes does not support istio + TLS cleanly : Flink assumes that pods will be able to communicate by ip-address, meanwhile istio + TLS does not allow routing by pod's ip-address.

This ticket is to track effort to support Flink with istio. Some workaround is to disable the istio sidecar container, but this is not secure https://doc.akka.io/docs/akka-management/current/bootstrap/istio.html. Akka allows to  secure the channel manually, but there is no documentation how to do this in the context of Flink. One potential solution for this is to have the documentation about how to configure akka cluster + TLS in flink.

For example when using native kubernetes deployment mode with high-availability (HA), and when new TaskManager pod is started to process a job, the TaskManager pod will attempt to register itself to the resource manager (JobManager). the TaskManager looks up the resource manager per ip-address (akka.tcp://flink@192.168.140.164:6123/user/rpc/resourcemanager_1).

Other affected features is metric collection.

Please see FLINK-31775 and FLINK-28171. Especially comment "Flink currently just doesn't support Istio."