Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-29495

Add ability to estimate per

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Duplicate
    • 2.4.4
    • None
    • ML
    • None

    Description

      In gensim, [the LDA model|https://radimrehurek.com/gensim/models/ldamodel.html] has a parameter eval_every that allows a user to specify that the model should be evaluated every X iterations to determine its log perplexity. This helps to determine convergence of the model, and whether or not the proper number of iterations has been chosen. Spark has no similar functionality in its implementation of LDA. This should be added, as it appears the only way to achieve this functionality would be to train models of varying numbers of iterations and evaluate each's log perplexity.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              csnardi Chris Nardi
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: