Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Not A Problem
-
2.3.0
-
None
-
None
Description
When calling ALS.fit() with the same seed on a dataset, the results (both the latent factors matrices and the accuracy of the recommendations) differ when we change the labels used to encode the users or items. The code example below illustrates this by just changing user ID 1 or 2 to an unused ID like 30. The user factors matrix changes, but not only the rows corresponding to users 1 or 2 but also the other rows.
Is this the intended behaviour?
val r = scala.util.Random
r.setSeed(123456)
val trainDataset1 = spark.sparkContext.parallelize(
(1 to 1000).map(_=> (r.nextInt(20), r.nextInt(100), r.nextInt(4) + 1)) // users go from 0 to 19
).toDF("user", "item", "rating")
val trainDataset2 = trainDataset1.withColumn("user", when(col("user")===1, 30).otherwise(col("user")))
val trainDataset3 = trainDataset1.withColumn("user", when(col("user")===2, 30).otherwise(col("user")))
val testDatasets = Array(trainDataset1, trainDataset2, trainDataset3).map(
_.groupBy("user").agg(collect_list("item").alias("watched"))
)
val Array(als1, als2, als3) = Array(trainDataset1, trainDataset2, trainDataset3).map(new ALS().setSeed(12345).fit(_))
als1.userFactors.show(5, false)
als2.userFactors.show(5, false)
als3.userFactors.show(5, false)
If we ask for recommendations and compare them with a test dataset also modified accordingly (in this example, the test dataset is exactly the train dataset) the results also differ:
val recommendations = Array(als1, als2, als3).map(x =>
x.recommendForAllUsers(20).map{
case Row(user: Int, recommendations: WrappedArray[Row]) => {
val items = recommendations.map{case Row(item: Int, score: Float) => item}
(user, items)
}
}.toDF("user", "recommendations")
)
val predictionsAndActualRDD = testDatasets.zip(recommendations).map{
case (testDataset, recommendationsDF) =>
testDataset.join(recommendationsDF, "user")
.rdd.map(r => {
(r.getAs[WrappedArray[Int]](r.fieldIndex("recommendations")).array,
r.getAs[WrappedArray[Int]](r.fieldIndex("watched")).array
)
})
}
val metrics = predictionsAndActualRDD.map(new RankingMetrics(_))
println(s"Precision at 5 of first model = ${metrics(0).precisionAt(5)}")
println(s"Precision at 5 of second model = ${metrics(1).precisionAt(5)}")
println(s"Precision at 5 of third model = ${metrics(2).precisionAt(5)}")
EDIT: The results also change if we just swap the IDs of some users, like:
val trainDataset4 = trainDataset1.withColumn("user", when(col("user")===1, 4) .when(col("user")===4, 1).otherwise(col("user")))