Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-6999

Encrypt Config Toolkit fails on very large flow.xml.gz files

    XMLWordPrintableJSON

Details

    Description

      A user reported failure when using the encrypt config toolkit to process (encrypt) a large flow.xml.gz. The compressed file was 49 MB, but was 687 MB uncompressed. It contained 545 encrypted values, and approximately 90 templates. This caused the toolkit to fail during loadFlowXml() unless the toolkit invocation set the heap to 8 GB via -Xms2g -Xmx8g. Even with the expanded heap, the serialization of the newly-encrypted flow XML to the file system fails with the following exception:

      Exception in thread "main" java.lang.OutOfMemoryError: Requested array size exceeds VM limit
      at java.lang.StringCoding.encode(StringCoding.java:350)
      at java.lang.String.getBytes(String.java:941)
      at org.apache.commons.io.IOUtils.write(IOUtils.java:1857)
      at org.apache.commons.io.IOUtils$write$0.call(Unknown Source)
      at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
      at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
      at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:141)
      at org.apache.nifi.properties.ConfigEncryptionTool$_writeFlowXmlToFile_closure5$_closure20.doCall(ConfigEncryptionTool.groovy:692)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
      at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
      at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
      at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1019)
      at groovy.lang.Closure.call(Closure.java:426)
      at groovy.lang.Closure.call(Closure.java:442)
      at org.codehaus.groovy.runtime.IOGroovyMethods.withCloseable(IOGroovyMethods.java:1622)
      at org.codehaus.groovy.runtime.NioGroovyMethods.withCloseable(NioGroovyMethods.java:1754)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.codehaus.groovy.runtime.metaclass.ReflectionMetaMethod.invoke(ReflectionMetaMethod.java:54)
      at org.codehaus.groovy.runtime.metaclass.NewInstanceMetaMethod.invoke(NewInstanceMetaMethod.java:56)
      at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274)
      at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
      at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
      at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
      at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
      at org.apache.nifi.properties.ConfigEncryptionTool$_writeFlowXmlToFile_closure5.doCall(ConfigEncryptionTool.groovy:691)
      

      The immediate fix was to remove the duplicated template definitions in the flow definition, returning the file to a reasonable size. However, if run as an inline replacement, this can cause the flow.xml.gz to be overwritten with an empty file, potentially leading to data loss. The following steps should be taken:

      1. Guard against loading/operating on/serializing large files (log statements, simple conditional checks)
      2. Handle large files internally (change from direct String access to BufferedInputStream, etc.)
      3. Document the internal memory usage of the toolkit in the toolkit guide
      4. Document best practices and steps to resolve issue in the toolkit guide

      Attachments

        Issue Links

          Activity

            People

              thenatog Nathan Gough
              alopresto Andy LoPresto
              Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 2h 40m
                  2h 40m