Details
-
New Feature
-
Status: Open
-
Minor
-
Resolution: Unresolved
-
1.17.0
-
None
-
None
Description
Currently in Python Flink API, when reading messages from a Kafka source, only SimpleStringSchema is available.
If the data is in arbitary binary format(e.g. marshalled Protocol Buffer msg) it may not be decodable with the default 'utf-8' encoding.
There's currently a workaround which is to manually set the encoding to 'ISO-8859-1' which supports all possible byte combinations.
However this is not an elegant solution.
We should support ByteArraySchema which outputs a raw byte array for subsequent unmarshalling.