Details
-
Bug
-
Status: Open
-
Not a Priority
-
Resolution: Unresolved
-
1.15.0
-
None
Description
For flink version 1.15.0, the configuration `external-resources` delimiter separation(";") functionality between external resource names is not working as expected.
Reproducing the bug setting up a flink cluster running using docker with two external resources named `abc` and `efg`:
docker-compose.yml:
services:
jobmanager:
image: flink:1.15.0-scala_2.12
ports:
- "8081:8081"
- "6123:6123"
command: jobmanager
environment:
- |
FLINK_PROPERTIES=
jobmanager.rpc.address: jobmanager
taskmanager:
image: flink:1.15.0-scala_2.12
depends_on:
- jobmanager
command: taskmanager
environment:
- |
FLINK_PROPERTIES=
jobmanager.rpc.address: jobmanager
taskmanager.numberOfTaskSlots: 8
external-resources: abc;efg
external-resource.abc.amount: 1
external-resource.efg.amount: 1
Related task manager logs:
root@2873c5ae72c2:/opt/flink# cat log/flink* | grep ExternalResourceUtils 2022-06-07 05:42:34,232 WARN org.apache.flink.runtime.externalresource.ExternalResourceUtils [] - The amount of the abc;efg should be configured. Will ignore that resource. 2022-06-07 05:42:34,232 INFO org.apache.flink.runtime.externalresource.ExternalResourceUtils [] - Enabled external resources: [] 2022-06-07 05:42:34,236 WARN org.apache.flink.runtime.externalresource.ExternalResourceUtils [] - Could not find driver class name for abc;efg. Please make sure external-resource.abc;efg.driver-factory.class is configured. 2022-06-07 05:42:34,240 WARN org.apache.flink.runtime.externalresource.ExternalResourceUtils [] - The amount of the abc;efg should be configured. Will ignore that resource.
Notice flink recognizes `abc;efg` as a single external resource rather than as `abc` and `efg`
The same issue does not exist in flink version 1.14.4
docker-compose.yml:
services:
jobmanager:
image: flink:1.14.4-scala_2.12
ports:
- "8081:8081"
- "6123:6123"
command: jobmanager
environment:
- |
FLINK_PROPERTIES=
jobmanager.rpc.address: jobmanager
taskmanager:
image: flink:1.14.4-scala_2.12
depends_on:
- jobmanager
command: taskmanager
environment:
- |
FLINK_PROPERTIES=
jobmanager.rpc.address: jobmanager
taskmanager.numberOfTaskSlots: 8
external-resources: abc;efg
external-resource.abc.amount: 1
external-resource.efg.amount: 1
related tm logs
docker logs ... | grep ExternalResourceUtils 2022-06-07 05:48:13,531 INFO org.apache.flink.runtime.externalresource.ExternalResourceUtils [] - Enabled external resources: [abc, efg] 2022-06-07 05:48:13,534 WARN org.apache.flink.runtime.externalresource.ExternalResourceUtils [] - Could not find driver class name for abc. Please make sure external-resource.abc.driver-factory.class is configured. 2022-06-07 05:48:13,534 WARN org.apache.flink.runtime.externalresource.ExternalResourceUtils [] - Could not find driver class name for efg. Please make sure external-resource.efg.driver-factory.class is configured.