Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
1.8.0
-
None
Description
Current implementation of PutAzureEventHub is too simplistic. It sends one flow file at a time synchronously which poses a significant bottlenec. It doesn't lift attributes into the message's User defined properties section. And most important problem is that you can't specify the partitioning key which plays critical role in a delivery consistency across multiple partitions in case there are more partitions. Without partitioning key multiple use-cases can't choose NiFi as an option.
Aim of this improvement is to consume up to specified amount of flow files, lift all attributes of each flow file into User defined properties section of each message and let processor to use one of those attributes as a partitioning key.
Each flow file in a batch is sent asynchronously which is way more faster then processor wait for all confirmations (or errors) and based on the result messages are transfered to success or failure output.
Attachments
Attachments
Issue Links
- links to