Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-25416

Hive metastore memory leak because datanucleus-api-jdo bug

    XMLWordPrintableJSON

Details

    Description

      I encountered a memory leak case. The MAT info :

      Full error message is :

      Cannot get Long result for param = 8 for column "`FUNCS`.`FUNC_ID`" : Operation not allowed after ResultSet closed

      This is because there is a bug in the JDOPersistenceManager.retrieveAll code.

      // code placeholder
      JDOPersistenceManager{
      public void retrieveAll(Collection pcs, boolean useFetchPlan) {
          this.assertIsOpen();
          ArrayList failures = new ArrayList();
          Iterator i = pcs.iterator();
      
          while(i.hasNext()) {
              try {
                  this.jdoRetrieve(i.next(), useFetchPlan);
              } catch (RuntimeException var6) {
                  failures.add(var6);
              }
          }
      
          if (!failures.isEmpty()) {
              throw new JDOUserException(Localiser.msg("010038"), (Exception[])((Exception[])failures.toArray(new Exception[failures.size()])));
          }
      }
      }
      

      In some extreme cases   the function of next() does not work . This will result in a very large failures ArrayList like as shown above.

       

      The bug detail can see this : https://github.com/datanucleus/datanucleus-api-jdo/issues/106

      This problem is fixed in datanucleus-api-jdo version 5.2.6. So we should upgrade it .

       

       

       

      Attachments

        1. leak.jpg
          483 kB
          shezm

        Issue Links

          Activity

            People

              Zing zzzzming95
              shezhiming shezm
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 2h 20m
                  2h 20m