Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-3368

"/kylin/kylin_metadata/metadata/" has many garbage for spark cubing

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • v2.2.0, v2.3.0
    • v2.4.0
    • Spark Engine
    • None

    Description

      If using Spark as cube engine, Kylin will dump metadata to HDFS in "/kylin/kylin_metadata/metadata/"; As time goes on, many files were left there.

       

      They should be deleted when the job is finished.

       

      hadoop fs -ls /kylin/kylin_metadata/metadata
      Found 583 items
      drwxr-xr-x - root hdfs 0 2018-02-11 16:20 /kylin/kylin_metadata/metadata/007d8b6c-5db1-478f-a373-7fd57231494b
      drwxr-xr-x - root hdfs 0 2018-02-21 08:24 /kylin/kylin_metadata/metadata/00f93335-0d85-4c19-953c-5e4660f4d58e
      drwxr-xr-x - root hdfs 0 2018-03-21 10:17 /kylin/kylin_metadata/metadata/011d3c7b-e707-42ca-b355-a0037820159b
      drwxr-xr-x - root hdfs 0 2018-04-07 00:41 /kylin/kylin_metadata/metadata/0263789f-969a-485f-aa6c-df4394a45df8
      drwxr-xr-x - root hdfs 0 2018-02-23 00:24 /kylin/kylin_metadata/metadata/02977825-92cd-475d-8a3f-0f75255a9ba9
      drwxr-xr-x - root hdfs 0 2018-02-15 08:24 /kylin/kylin_metadata/metadata/02e058fd-ac6a-451f-99b4-f0b03ad6e3a5
      drwxr-xr-x - root hdfs 0 2018-03-15 10:21 /kylin/kylin_metadata/metadata/0337c6c9-aebb-4ae0-8485-9905283fc818
      drwxr-xr-x - root hdfs 0 2018-02-23 12:22 /kylin/kylin_metadata/metadata/03d7f34f-10dc-4329-9dde-f646cc040ba9
      drwxr-xr-x - root hdfs 0 2018-04-10 12:20 /kylin/kylin_metadata/metadata/03fa7654-7b14-4bda-aed8-64ac9d08081a
      drwxr-xr-x - root hdfs 0 2018-03-30 00:23 /kylin/kylin_metadata/metadata/0406e5bb-112f-4c54-bdf8-7b1e34652522
      drwxr-xr-x - root hdfs 0 2018-03-24 00:33 /kylin/kylin_metadata/metadata/0426907d-541f-4f23-9db7-9a965f5b6739
      drwxr-xr-x - root hdfs 0 2018-02-07 15:03 /kylin/kylin_metadata/metadata/04771302-2090-4793-956d-bf5ba9360d9b
      drwxr-xr-x - root hdfs 0 2018-03-30 00:20 /kylin/kylin_metadata/metadata/0482b928-6b09-48d1-909f-68c66f71a827
      drwxr-xr-x - root hdfs 0 2018-03-04 16:36 /kylin/kylin_metadata/metadata/04943bbd-8425-4304-b9c8-cc13a55e8ac1
      drwxr-xr-x - root hdfs 0 2018-03-05 08:35 /kylin/kylin_metadata/metadata/04ee2c3d-bd07-4e7a-90db-3d87d088ff7d
      drwxr-xr-x - root hdfs 0 2018-03-02 16:24 /kylin/kylin_metadata/metadata/05808a9d-2e55-4dc3-8c28-5b662068dda8
      drwxr-xr-x - root hdfs 0 2018-03-04 08:35 /kylin/kylin_metadata/metadata/05c63d7c-4ea5-48fc-897d-6db9d78ffaf7
      drwxr-xr-x - root hdfs 0 2018-03-01 01:02 /kylin/kylin_metadata/metadata/05ea3568-e85f-452a-b44d-4657145a724a

      Attachments

        Issue Links

          Activity

            People

              shaofengshi Shao Feng Shi
              shaofengshi Shao Feng Shi
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: