Details
-
Test
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
3.0.0
-
None
Description
The LogAppender has a limit for the number of logged events. By default, it is 100 entries. All tests that use LogAppender finish successfully with this limit but sometimes one of the tests fails with the exception like:
java.lang.IllegalStateException: Number of events reached the limit of 100 while logging CSV header matches to schema w/ enforceSchema. at org.apache.spark.SparkFunSuite$LogAppender.append(SparkFunSuite.scala:200) at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251) at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66) at org.apache.log4j.Category.callAppenders(Category.java:206) at org.apache.log4j.Category.forcedLog(Category.java:391) at org.apache.log4j.Category.log(Category.java:856)
For example, the CSV test "SPARK-23786: warning should be printed if CSV header doesn't conform to schema" uses 2 log appenders. The `success_trace.txt` contains stored log event in normal runs, another one `fail_trace.txt` which I got while re-running the test in a loop.
The difference is in:
[35] Block broadcast_222 stored as values in memory (estimated size 286.9 KiB, free 1998.5 MiB) [36] Removed broadcast_214_piece0 on 192.168.1.66:52354 in memory (size: 5.7 KiB, free: 2003.8 MiB) [37] Removed broadcast_204_piece0 on 192.168.1.66:52354 in memory (size: 5.7 KiB, free: 2003.9 MiB) [38] Removed broadcast_200_piece0 on 192.168.1.66:52354 in memory (size: 3.7 KiB, free: 2003.9 MiB) [39] Removed broadcast_207_piece0 on 192.168.1.66:52354 in memory (size: 24.2 KiB, free: 2003.9 MiB) [40] Removed broadcast_208_piece0 on 192.168.1.66:52354 in memory (size: 24.2 KiB, free: 2003.9 MiB) [41] Removed broadcast_194_piece0 on 192.168.1.66:52354 in memory (size: 5.7 KiB, free: 2003.9 MiB) [42] Removed broadcast_216_piece0 on 192.168.1.66:52354 in memory (size: 53.3 KiB, free: 2004.0 MiB) [43] Removed broadcast_195_piece0 on 192.168.1.66:52354 in memory (size: 3.7 KiB, free: 2004.0 MiB) [44] Removed broadcast_219_piece0 on 192.168.1.66:52354 in memory (size: 5.7 KiB, free: 2004.0 MiB) [45] Removed broadcast_198_piece0 on 192.168.1.66:52354 in memory (size: 24.2 KiB, free: 2004.0 MiB) [46] Removed broadcast_201_piece0 on 192.168.1.66:52354 in memory (size: 53.4 KiB, free: 2004.0 MiB) [47] Removed broadcast_205_piece0 on 192.168.1.66:52354 in memory (size: 3.7 KiB, free: 2004.0 MiB) [48] Removed broadcast_197_piece0 on 192.168.1.66:52354 in memory (size: 24.2 KiB, free: 2004.1 MiB) [49] Removed broadcast_189_piece0 on 192.168.1.66:52354 in memory (size: 5.7 KiB, free: 2004.1 MiB) [50] Removed broadcast_193_piece0 on 192.168.1.66:52354 in memory (size: 24.2 KiB, free: 2004.1 MiB) [51] Removed broadcast_196_piece0 on 192.168.1.66:52354 in memory (size: 53.3 KiB, free: 2004.2 MiB) [52] Removed broadcast_206_piece0 on 192.168.1.66:52354 in memory (size: 53.4 KiB, free: 2004.2 MiB) [53] Removed broadcast_209_piece0 on 192.168.1.66:52354 in memory (size: 5.7 KiB, free: 2004.2 MiB) [54] Removed broadcast_192_piece0 on 192.168.1.66:52354 in memory (size: 24.2 KiB, free: 2004.2 MiB) [55] Removed broadcast_215_piece0 on 192.168.1.66:52354 in memory (size: 3.7 KiB, free: 2004.2 MiB) [56] Removed broadcast_210_piece0 on 192.168.1.66:52354 in memory (size: 3.7 KiB, free: 2004.2 MiB) [57] Removed broadcast_199_piece0 on 192.168.1.66:52354 in memory (size: 5.7 KiB, free: 2004.2 MiB) [58] Removed broadcast_221_piece0 on 192.168.1.66:52354 in memory (size: 53.4 KiB, free: 2004.3 MiB) [59] Removed broadcast_213_piece0 on 192.168.1.66:52354 in memory (size: 24.2 KiB, free: 2004.3 MiB) [60] Removed broadcast_191_piece0 on 192.168.1.66:52354 in memory (size: 53.4 KiB, free: 2004.4 MiB) [61] Removed broadcast_220_piece0 on 192.168.1.66:52354 in memory (size: 3.7 KiB, free: 2004.4 MiB) [62] Block broadcast_222_piece0 stored as bytes in memory (estimated size 24.2 KiB, free 2002.0 MiB) [63] Added broadcast_222_piece0 in memory on 192.168.1.66:52354 (size: 24.2 KiB, free: 2004.4 MiB) [64] Removed broadcast_203_piece0 on 192.168.1.66:52354 in memory (size: 24.2 KiB, free: 2004.4 MiB) [65] Created broadcast 222 from apply at OutcomeOf.scala:85 [66] Removed broadcast_218_piece0 on 192.168.1.66:52354 in memory (size: 24.2 KiB, free: 2004.4 MiB) [67] Removed broadcast_211_piece0 on 192.168.1.66:52354 in memory (size: 53.4 KiB, free: 2004.5 MiB) [68] Removed broadcast_190_piece0 on 192.168.1.66:52354 in memory (size: 3.7 KiB, free: 2004.5 MiB) [69] Block broadcast_223 stored as values in memory (estimated size 286.9 KiB, free 2002.5 MiB) [70] Removed broadcast_212_piece0 on 192.168.1.66:52354 in memory (size: 24.2 KiB, free: 2004.5 MiB) [71] Removed broadcast_188_piece0 on 192.168.1.66:52354 in memory (size: 24.2 KiB, free: 2004.5 MiB) [72] Removed broadcast_187_piece0 on 192.168.1.66:52354 in memory (size: 24.2 KiB, free: 2004.5 MiB) [73] Removed broadcast_202_piece0 on 192.168.1.66:52354 in memory (size: 24.2 KiB, free: 2004.6 MiB) [74] Removed broadcast_217_piece0 on 192.168.1.66:52354 in memory (size: 24.2 KiB, free: 2004.6 MiB) [75] Block broadcast_223_piece0 stored as bytes in memory (estimated size 24.2 KiB, free 2004.0 MiB) [76] Added broadcast_223_piece0 in memory on 192.168.1.66:52354 (size: 24.2 KiB, free: 2004.6 MiB) [77] Created broadcast 223 from apply at OutcomeOf.scala:85 [78] Starting job: apply at OutcomeOf.scala:85 [79] Got job 134 (apply at OutcomeOf.scala:85) with 1 output partitions [80] Final stage: ResultStage 134 (apply at OutcomeOf.scala:85) [81] Parents of final stage: List() [82] Missing parents: List() [83] Submitting ResultStage 134 (MapPartitionsRDD[809] at apply at OutcomeOf.scala:85), which has no missing parents [84] Block broadcast_224 stored as values in memory (estimated size 11.0 KiB, free 2004.0 MiB) [85] Block broadcast_224_piece0 stored as bytes in memory (estimated size 5.7 KiB, free 2004.0 MiB) [86] Added broadcast_224_piece0 in memory on 192.168.1.66:52354 (size: 5.7 KiB, free: 2004.5 MiB) [87] Created broadcast 224 from broadcast at DAGScheduler.scala:1200 [88] Submitting 1 missing tasks from ResultStage 134 (MapPartitionsRDD[809] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) [89] Adding task set 134.0 with 1 tasks [90] Starting task 0.0 in stage 134.0 (TID 134, 192.168.1.66, executor driver, partition 0, PROCESS_LOCAL, 8029 bytes) [91] Running task 0.0 in stage 134.0 (TID 134)
Attachments
Attachments
Issue Links
- links to