Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-23149

hbase shouldPerformMajorCompaction logic is not correct

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.4.9
    • 1.6.0, 1.4.12, 1.3.7
    • Compaction
    • None

    Description

      we can know that the major compaction is skipping  from the below regionserver's log, but it is compacting that region.

      and read the code and find it is not correct  and i add mark  "/*** ***/" below

       

      public boolean shouldPerformMajorCompaction(final Collection<StoreFile> filesToCompact)
      throws IOException {
      if (lowTimestamp > 0L && lowTimestamp < (now - mcTime)) {
      if (filesToCompact.size() == 1) {
      if (sf.isMajorCompaction() && (cfTTL == Long.MAX_VALUE || oldest < cfTTL)) {
      float blockLocalityIndex =
      sf.getHDFSBlockDistribution().getBlockLocalityIndex(
      RSRpcServices.getHostname(comConf.conf, false));
      if (blockLocalityIndex < comConf.getMinLocalityToForceCompact())

      { result = true; }

      else

      { LOG.debug("Skipping major compaction of " + regionInfo + " because one (major) compacted file only, oldestTime " + oldest + "ms is < TTL=" + cfTTL + " and blockLocalityIndex is " + blockLocalityIndex + " (min " + comConf.getMinLocalityToForceCompact() + ")"); }

      } else if (cfTTL != HConstants.FOREVER && oldest > cfTTL)

      { LOG.debug("Major compaction triggered on store " + regionInfo + ", because keyvalues outdated; time since last major compaction " + (now - lowTimestamp) + "ms"); result = true; }

      } else

      { LOG.debug("Major compaction triggered on store " + regionInfo + "; time since last major compaction " + (now - lowTimestamp) + "ms"); }

      result = true;  /****here it is not right, it should be move to the above *****/
      }
      return result;
      }

       

      2019-09-27 09:09:35,960 DEBUG [st129,16020,1568236573216_ChoreService_1] compactions.RatioBasedCompactionPolicy: Skipping major compaction of 100E_POINT_point_2ddata_z3_geom_GpsTime_v6,\x17,1568215725799.413a563092544e8df480fd601b2de71b. because one (major) compacted file only, oldestTime 3758085589ms is < TTL=9223372036854775807 and blockLocalityIndex is 1.0 (min 0.0)
      2019-09-27 09:09:35,961 DEBUG [st129,16020,1568236573216_ChoreService_1] compactions.SortedCompactionPolicy: Selecting compaction from 1 store files, 0 compacting, 1 eligible, 100 blocking
      2019-09-27 09:09:35,961 DEBUG [st129,16020,1568236573216_ChoreService_1] regionserver.HStore: 413a563092544e8df480fd601b2de71b - d: Initiating major compaction (all files)
      2019-09-27 09:09:35,961 DEBUG [st129,16020,1568236573216_ChoreService_1] regionserver.CompactSplitThread: Large Compaction requested: org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@4b5582f1; Because: CompactionChecker requests major compaction; use default priority; compaction_queue=(1:0), split_queue=0, merge_queue=0
      2019-09-27 09:09:35,961 INFO [regionserver/st129/10.3.72.129:16020-longCompactions-1568236575579] regionserver.HRegion: Starting compaction on d in region 100E_POINT_point_2ddata_z3_geom_GpsTime_v6,\x17,1568215725799.413a563092544e8df480fd601b2de71b.
      2019-09-27 09:09:35,961 INFO [regionserver/st129/10.3.72.129:16020-longCompactions-1568236575579] regionserver.HStore: Starting compaction of 1 file(s) in d of 100E_POINT_point_2ddata_z3_geom_GpsTime_v6,\x17,1568215725799.413a563092544e8df480fd601b2de71b. into tmpdir=hdfs://st129:8020/hbase/data/default/100E_POINT_point_2ddata_z3_geom_GpsTime_v6/413a563092544e8df480fd601b2de71b/.tmp, totalSize=5.1 G
      2019-09-27 09:09:35,961 DEBUG [regionserver/st129/10.3.72.129:16020-longCompactions-1568236575579] compactions.Compactor: Compacting hdfs://st129:8020/hbase/data/default/100E_POINT_point_2ddata_z3_geom_GpsTime_v6/413a563092544e8df480fd601b2de71b/d/3b4080f9b6f149e1b0a476058c8564e6, keycount=83914030, bloomtype=NONE, size=5.1 G, encoding=FAST_DIFF, compression=SNAPPY, seqNum=2621061, earliestPutTs=1565788490371

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            jackylau Jacky Lau
            jackylau Jacky Lau
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment