Uploaded image for project: 'ORC'
  1. ORC
  2. ORC-653

OrcFile#getStaticMemoryManager caches initial configuration and leaks classloaders

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.5.6
    • None
    • None
    • None

    Description

      As part of our effort to find classloader leaks in Apache Flink, we found the following issue coming from OrcFile.

        private static synchronized MemoryManager getStaticMemoryManager(
            final Configuration conf) {
          if (memoryManager == null) {
            memoryManager = new ThreadLocal<MemoryManager>() {
              @Override
              protected MemoryManager initialValue() {
                return new MemoryManagerImpl(conf);
              }
            };
          }
          return memoryManager.get();
        }
      

      Here the original conf will be used for all other memory managers in the future. If you close the classloader of that conf, future usages of any MemoryManager coming through that ThreadLocal will fail.

      For our use case, where there is only one conf/MemoryManager per Thread, it would be sufficient, to explicitly initialize the thread locals.

        private static synchronized MemoryManager getStaticMemoryManager(
            final Configuration conf) {
          MemoryManager manager = memoryManager.get();
          if (manager == null) {
            manager = new MemoryManagerImpl(conf);
            memoryManager.set(manager);
          }
          return manager;
        }
      

      I can provide a patch and additional information.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              arvid Arvid Heise
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: