Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-4494

IDFModel.transform() add support for single vector

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 1.1.1, 1.2.0
    • 1.3.0
    • MLlib
    • None

    Description

      For now when using the tfidf implementation of mllib you have no other possibility to map your data back onto i.e. labels or ids than use a hackish way with ziping:

      1. Persist input RDD. 2. Transform it to just vectors and apply IDFModel 3. zip with original RDD 4. transform label and new vector to LabeledPoint

      Source:http://stackoverflow.com/questions/26897908/spark-mllib-tfidf-implementation-for-logisticregression

      I think as in production alot of users want to map their data back to some identifier, it would be a good imporvement to allow using a single vector on IDFModel.transform()

      Attachments

        Activity

          People

            yuu.ishikawa@gmail.com Yu Ishikawa
            Jean-Philippe Quemener Jean-Philippe Quemener
            Xiangrui Meng Xiangrui Meng
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: