Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Duplicate
-
SystemML 1.0.0
-
None
Description
Conditionals in nn layers introduce transient read/write variables that disables fused operators such as CP relu_maxpooling_backward and hence redundant execute sparsity-introducing sel+ operator. This operator causes unnecessary dense-to-sparse-to-dense conversion and becomes the heavy hitter after native BLAS change. Note: some fused operators such as CP relu_maxpooling are still applied because there is no conditional in between those layers.
Without conditionals in dropout layer: https://github.com/apache/incubator-systemml/blob/master/scripts/nn/layers/dropout.dml#L49-L53
Iter:2000.0, training loss:0.003149394810197065, training accuracy:100.0 Iter:2000.0, validation loss:191.9888157354513, validation accuracy:96.875 SystemML Statistics: Total elapsed time: 416.609 sec. Total compilation time: 0.000 sec. Total execution time: 416.609 sec. Number of compiled Spark inst: 69. Number of executed Spark inst: 2. Native mkl calls (LibMatrixMult/LibMatrixDNN): 4270/10553. Cache hits (Mem, WB, FS, HDFS): 277973/0/0/0. Cache writes (WB, FS, HDFS): 143616/0/0. Cache times (ACQr/m, RLS, EXP): 0.101/0.080/1.988/0.000 sec. HOP DAGs recompiled (PRED, SB): 0/2277. HOP DAGs recompile time: 6.146 sec. Spark ctx create time (lazy): 0.027 sec. Spark trans counts (par,bc,col):0/0/0. Spark trans times (par,bc,col): 0.000/0.000/0.000 secs. Total JIT compile time: 37.746 sec. Total JVM GC count: 3949. Total JVM GC time: 56.609 sec. Heavy hitter instructions (name, time, count): -- 1) conv2d_bias_add 48.984 sec 4514 -- 2) conv2d_backward_filter 47.780 sec 4026 -- 3) -* 38.246 sec 16104 -- 4) +* 35.902 sec 8052 -- 5) + 34.227 sec 30566 -- 6) ba+* 30.643 sec 12566 -- 7) relu_maxpooling_backward 29.678 sec 4026 -- 8) conv2d_backward_data 28.520 sec 2013 -- 9) * 26.825 sec 35275 -- 10) relu_backward 24.842 sec 6039
With conditional, we add sel+ to the heavy hitter:
-- 1) sel+ 55.054 sec 6283
mwdusenb@us.ibm.com Since you created the layers, I think you should decide how best to restructure the DML. My recommendation would be to create two layers in case of conditionals.