Parallelism seems to cause difference in CrossValidation model metrics

I can only reproduce this issue when running Spark on different Amazon EMR versions, but it seems that between Spark 2.3.1 and 2.3.2 (corresponding to EMR versions 5.17/5.18) the presence of the parallelism parameter was causing AUC metric to increase. Literally, I run the same exact code with and without parallelism and the AUC of my models (logistic regression) are changing significantly. I can't find a previous bug report relating to this, so I'm posting this as new.