Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27447

Add collaborate filtering Explain API in SPARKML

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0
    • Fix Version/s: None
    • Component/s: ML
    • Labels:
      None

      Description

      Machine learning recommender systems have supercharged the online retail environment by directly targeting what the customer wants. While customers are getting better product recommendations than ever before, in the age of GDPR there is growing concern about customer privacy and transparency with ML models. Many are asking, just why am I receiving these recommendations? While the current Implicit Collaborative Filtering (CF) algorithm in spark.ml is great for generating recommendations at scale, its currently lacks any method to explain why a particular customer is getting the recommendations they are getting. In this talk, we demonstrate a way to expand collaborative filtering so that the viewing history of a customer can be directly related to their recommendations. Why were you recommended footwear? Well, 40% of this recommendation came from browsing runners and 20% came from the shorts you recently purchased. Turns out, rethinking of the linear algebra in the current spark.ml CF implementation makes this possible. We show how this is done and demonstrate its implemented as a new feature to spark.ml, expanding the API to allow everyone to explain recommendations at scale and create a more transparent ML future.

       

       

      This project is going to present in Spark summit 2019:
      https://databricks.com/sparkaisummit/north-america/sessions-single-2019?id=56

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              guohao guohao xiao
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: