Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-29818 Missing persist on RDD
  3. SPARK-29813

Missing persist in mllib.PrefixSpan.findFrequentItems()

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 2.4.3
    • None
    • MLlib
    • None

    Description

      There are three actions in this piece of code: reduceByKey, sortBy, and collect. But data is not persisted, which will cause recomputation.

        private[fpm] def findFrequentItems[Item: ClassTag](
            data: RDD[Array[Array[Item]]],
            minCount: Long): Array[Item] = {
      
          data.flatMap { itemsets =>
            val uniqItems = mutable.Set.empty[Item]
            itemsets.foreach(set => uniqItems ++= set)
            uniqItems.toIterator.map((_, 1L))
          }.reduceByKey(_ + _).filter { case (_, count) =>
            count >= minCount
          }.sortBy(-_._2).map(_._1).collect()
        }
      

      This issue is reported by our tool CacheCheck, which is used to dynamically detecting persist()/unpersist() api misuses.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              spark_cachecheck IcySanwitch
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: