Details
-
New Feature
-
Status: Resolved
-
Minor
-
Resolution: Won't Fix
-
3.0.0
-
None
-
None
Description
This feature enables Maximum A Posteriori (MAP) optimization for Logistic Regression based on a Gaussian prior. In practice, this is just implementing a more general form of L2 regularization parameterized by a (multivariate) mean and precisions (inverse of variance) vectors.
Prior regularization is calculated through the following formula:
where:
- λ: regularization parameter (regParam)
- K: number of coefficients (weights vector length)
- w~i~ with prior Normal(μ~i~, β~i~2)
Reference: Bishop, Christopher M. (2006). Pattern Recognition and Machine Learning (section 4.5). Berlin, Heidelberg: Springer-Verlag.
Existing implementations
- Python: bayes_logistic
Implementation
- 2 new parameters added to LogisticRegression: priorMean and priorPrecisions.
- 1 new class (PriorRegularization) implements the calculations of the value and gradient of the prior regularization term.
- Prior regularization is enabled when both vectors are provided and regParam > 0 and elasticNetParam < 1.
Tests
- DifferentiableRegularizationSuite
- Prior regularization
- LogisticRegressionSuite
- prior precisions should be required when prior mean is set
- prior mean should be required when prior precisions is set
- `regParam` should be positive when using prior regularization
- `elasticNetParam` should be less than 1.0 when using prior regularization
- prior mean and precisions should have equal length
- priors' length should match number of features
- binary logistic regression with prior regularization equivalent to L2
- binary logistic regression with prior regularization equivalent to L2 (bis)
- binary logistic regression with prior regularization