Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-12339

Sensitive Dynamic Properties not properly decrypted, resulting in wrong property value and ever-growing flow.json.gz

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • None
    • 2.0.0-M1
    • Core Framework
    • None

    Description

      To replication, create an InvokeHTTP Processor. Add a Sensitive Dynamic Property named "Authorization" with a value of "Bearer fsi8y3ofysp9f8ncp9nupnu8p3s9nu3s9" (it's ok that the value is nonsense). Apply the changes.

      Check the flow.json.gz:

      cat conf/flow.json.gz | gunzip - | jq | grep Authorization

      Restart NiFi.

      The value is no longer correct. And if you run the cat command above, you'll see the value has doubled in length. After restarting several times we can see this:

      nifi-2.0.0-SNAPSHOT $ cat conf/flow.json.gz | gunzip - | jq | grep Authorization
                    "Authorization": "enc{f1f9ba180c6468ff8ce393955034e69383739de54b44ef42b1bf2050c2639e83815d940b8a0cf9f5bc65bdf36f7df59bff9d7e69fa02f0ccc25c8b381684550c8fc6b6a8c570998064ef730f05b0dc}",
      
      -- restart --nifi-2.0.0-SNAPSHOT $ cat conf/flow.json.gz | gunzip - | jq | 
      
      grep Authorization
                    "Authorization": "enc{e4455b884d07a7156397d2f60ce3a2f44be909084403f5a84af205bae2af6dbfa2adf47a33d6663799ab523915e9323064554030236b928d5b1684b0a9d635b6589d878b731c35ae1560fbef5627a433b23fb331657e66af355ac356a1c9cd1435c0836a4ecb872966c2852aa3b13e179da1a0f7898c64173b27363458c01dbf7c8595a5dfe9ab798834568c9e0a52fefaf03f6f9d1bdf6ad230fea7cf1e8663a78a6b964d945c729d9ae678e2eaba8910d02373cd9acd08e7a047e0c676ee8a13e9c0}",
      
      -- restart --nifi-2.0.0-SNAPSHOT $ cat conf/flow.json.gz | gunzip - | jq | 
      
      grep Authorization
                    "Authorization": "enc{1aeb6970c1ff7f10b88f5b94a2c0cfa70c179638eb976ff7580f5b2546a64b4d96ae834afff9d01cae79c98b9ca4d73af604eab5e95013047e79c152d3e90b3c556e054f9478713eb156da41477d59668902c606f3f300e9804b8a504712822b5f072a5a596c2ba1706520f0163ce8bf0a51dbaf84ee9359c60e55df029dec700725ff1ac599774d4271d5c390ad49d4b350d21bee9f2c235a81f5356d85279db7b4e335bc11fc0d6bf1045a6d2610ff61d8b9da931fc026d356a3d9a9b738312d283c01740757a286e5eb9ad675daa14a391d3df694eaeeb6c66085976a88c86a08052b3eb046e622e5346205bc1e38bfe4aed2ff130595688e4b72d217f29a5c24a28bc06c7bb55e4fd2d25fea15ce523e92b8d721e9a9c0d08ab6d1634cb027658c868feacd89462796b604db7dc55cc2bba7c650f77148bad4ec7328ae8dbeed743420b5b640061f36ed8c8c1db200bbe6a241d6eb370cb024a5881fc734d722e2f1091f1ffa178ad841a4859c9dc734b66a628fbfeb8c3f0a1e5d02e28ce3e2c04737ab5b92d032fafe21ebe5abd542731228b394356bb5b547c68517f972864351022d2ef1118426}",
      
      -- restart --
      
      nifi-2.0.0-SNAPSHOT $ cat conf/flow.json.gz | gunzip - | jq | grep Authorization
                    "Authorization": "enc{7446029842eae314646fb90250ed4d9e2425bb81694a91408623e2a49f987df74bd8e08e1996b955f70bcc1a961a7356025ee6459bde7d7cec2235e66ff88cb2addb5365c4d46a3df0684d392cd170e2a1105ab6f7e05d03b90f8e6b50fedd904f1043ff13b49817f70b316c89978ca4d4c3a137627728ebd576aa72250d2716915a1ed1aa587ae6dde8a2e560482c3bd33c527904e9a06b92e52e55349bf6907834fc14cc0cbf3c3002def88e1e8b7f05f1302ceece2659fa4f9b4dfdbd7f3baf140a7be105671a17a51e5c15c09a2c233f7255d78ca7761402fbf0053f0e0f108231c1cef427315d5d2fd60f5de78d44c8e6ee159d2fe702a946ab3cf404f96146920e95aedd186a9db85b216eb582d56d899fe7a91df801e95759c05580c263a4870ebe75304fed8484388a7b1706dd6ece7404aae044df57508bb837ac36c9a4118d0efd5b0661ea765485af26f11bf39cc0ee3450dc6533c1adc2205321035107895bc4408183e14b167b2a0a4e3cdf95d1a8742325067c059c96ca21a2ca14a36212f2dd47eef394a7f93db050a14bc561638aa667e01ac1a801de3f9e51ef157eb886a2e99a02834f618dafd56f72a469d4f51a472d0aff632b697696fb50ee47cdc7bb4b0c0dbc4135ca58e8562f3308c0d45b8c983c38967f8cb45b7f82e01d5ce291ca7001150a2e630939db902135cf6249ebfa3319b72b46fcbed82cf30dd3a6a2318a3f4c1cb5f1b90c821b4df1caf5777bc85c6e637dcc9b397aca2f29cc2f6fc6ca227a2a7692a66c88f3e719e0cfe05f6aaef79a99e1336f90c8886c9c30917577a320818f6bd0f790f0f145181bf665232224fbb6094efa3b1782869c714babdc196f2cf253f3f1b112d59f9306c71bd909d7fc3a895739f1731c2831f3bb65d1847583a99fada5f77c00bd433776a6c3edbf837059be26ad5105f175b0e81ea58fc6092ff0593c882c713e6b2e6464cb063848866a594c1e992d5fdf121e4a26e844512d21012bf4fe3dba1cfd3cdd9200ec034301643f79a1eec65b71f4c17d3cccf63c779736fb741dfc61a661c46b63e2bb74209164c6f7b060e311a3c954198b32d19fd7751485b840931dfb913411b4c3b6b3b05e8c886af1518cc41ef31f0f3af7c8e6ef0adcb1dd769657a51c5c52dd495b9f1b3d2f86bb70d86dc94f7bb764eb65fda0c28f0bf7f0f990a1f8b382b37f976184ccbc315bf25ed2eedf5cf75b6d9f07a12d3c5ca04a4c549692ef20}",

      Eventually this caused OOME on my NiFi instance. When checking the flow.json.gz the file was 70 MB! After removing that property value, the size shrank to 29 KB.

      Attachments

        Issue Links

          Activity

            People

              exceptionfactory David Handermann
              markap14 Mark Payne
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 0.5h
                  0.5h