Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Incomplete
-
2.2.0
-
None
-
AWS EMR 5.8.0
Spark 2.2.0
Description
When I try to read ~9600 Json files using
val test = spark.read.option("header", true).option("inferSchema", true).json(paths: _*)
Any action on the above created data frame results in:
Caused by: org.codehaus.janino.JaninoRuntimeException: Code of method "apply2_1$(Lorg/apache/spark/sql/catalyst/expressions/GeneratedClass$SpecificUnsafeProjection;Lorg/apache/spark/sql/catalyst/InternalRow;)V" of class "org.apache.spark.sql.catalyst.expressions.Generat[73/1850] pecificUnsafeProjection" grows beyond 64 KB at org.codehaus.janino.CodeContext.makeSpace(CodeContext.java:949) at org.codehaus.janino.CodeContext.write(CodeContext.java:839) at org.codehaus.janino.UnitCompiler.writeOpcode(UnitCompiler.java:11081) at org.codehaus.janino.UnitCompiler.compileGet2(UnitCompiler.java:4546) at org.codehaus.janino.UnitCompiler.access$7500(UnitCompiler.java:206) at org.codehaus.janino.UnitCompiler$12.visitMethodInvocation(UnitCompiler.java:3774) at org.codehaus.janino.UnitCompiler$12.visitMethodInvocation(UnitCompiler.java:3762) at org.codehaus.janino.Java$MethodInvocation.accept(Java.java:4328) at org.codehaus.janino.UnitCompiler.compileGet(UnitCompiler.java:3762) at org.codehaus.janino.UnitCompiler.compileGetValue(UnitCompiler.java:4933) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:3180) at org.codehaus.janino.UnitCompiler.access$5000(UnitCompiler.java:206) at org.codehaus.janino.UnitCompiler$9.visitMethodInvocation(UnitCompiler.java:3151) at org.codehaus.janino.UnitCompiler$9.visitMethodInvocation(UnitCompiler.java:3139) at org.codehaus.janino.Java$MethodInvocation.accept(Java.java:4328) at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:3139) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:2112) at org.codehaus.janino.UnitCompiler.access$1700(UnitCompiler.java:206) at org.codehaus.janino.UnitCompiler$6.visitExpressionStatement(UnitCompiler.java:1377) at org.codehaus.janino.UnitCompiler$6.visitExpressionStatement(UnitCompiler.java:1370) at org.codehaus.janino.Java$ExpressionStatement.accept(Java.java:2558) at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1370) at org.codehaus.janino.UnitCompiler.compileStatements(UnitCompiler.java:1450) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:1436) at org.codehaus.janino.UnitCompiler.access$1600(UnitCompiler.java:206) at org.codehaus.janino.UnitCompiler$6.visitBlock(UnitCompiler.java:1376) at org.codehaus.janino.UnitCompiler$6.visitBlock(UnitCompiler.java:1370) at org.codehaus.janino.Java$Block.accept(Java.java:2471) at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1370) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:2220) at org.codehaus.janino.UnitCompiler.access$1800(UnitCompiler.java:206) at org.codehaus.janino.UnitCompiler$6.visitIfStatement(UnitCompiler.java:1378) at org.codehaus.janino.UnitCompiler$6.visitIfStatement(UnitCompiler.java:1370) at org.codehaus.janino.Java$IfStatement.accept(Java.java:2621) at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1370) at org.codehaus.janino.UnitCompiler.compileStatements(UnitCompiler.java:1450) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:1436) at org.codehaus.janino.UnitCompiler.access$1600(UnitCompiler.java:206) at org.codehaus.janino.UnitCompiler$6.visitBlock(UnitCompiler.java:1376) at org.codehaus.janino.UnitCompiler$6.visitBlock(UnitCompiler.java:1370) at org.codehaus.janino.Java$Block.accept(Java.java:2471) at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1370) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:2220) at org.codehaus.janino.UnitCompiler.access$1800(UnitCompiler.java:206) at org.codehaus.janino.UnitCompiler$6.visitIfStatement(UnitCompiler.java:1378) at org.codehaus.janino.UnitCompiler$6.visitIfStatement(UnitCompiler.java:1370) at org.codehaus.janino.Java$IfStatement.accept(Java.java:2621) at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1370) at org.codehaus.janino.UnitCompiler.compileStatements(UnitCompiler.java:1450) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:1436) at org.codehaus.janino.UnitCompiler.access$1600(UnitCompiler.java:206) at org.codehaus.janino.UnitCompiler$6.visitBlock(UnitCompiler.java:1376) at org.codehaus.janino.UnitCompiler$6.visitBlock(UnitCompiler.java:1370) at org.codehaus.janino.Java$Block.accept(Java.java:2471) at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1370) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:1538) at org.codehaus.janino.UnitCompiler.access$1900(UnitCompiler.java:206) at org.codehaus.janino.UnitCompiler$6.visitForStatement(UnitCompiler.java:1379) at org.codehaus.janino.UnitCompiler$6.visitForStatement(UnitCompiler.java:1370) at org.codehaus.janino.Java$ForStatement.accept(Java.java:2660) at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1370) at org.codehaus.janino.UnitCompiler.compileStatements(UnitCompiler.java:1450) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:1436) at org.codehaus.janino.UnitCompiler.access$1600(UnitCompiler.java:206) at org.codehaus.janino.UnitCompiler$6.visitBlock(UnitCompiler.java:1376) at org.codehaus.janino.UnitCompiler$6.visitBlock(UnitCompiler.java:1370) at org.codehaus.janino.Java$Block.accept(Java.java:2471) at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1370) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:1538) at org.codehaus.janino.UnitCompiler.access$1900(UnitCompiler.java:206) at org.codehaus.janino.UnitCompiler$6.visitForStatement(UnitCompiler.java:1379) at org.codehaus.janino.UnitCompiler$6.visitForStatement(UnitCompiler.java:1370) at org.codehaus.janino.Java$ForStatement.accept(Java.java:2660) at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1370) at org.codehaus.janino.UnitCompiler.compileStatements(UnitCompiler.java:1450) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:1436) at org.codehaus.janino.UnitCompiler.access$1600(UnitCompiler.java:206) at org.codehaus.janino.UnitCompiler$6.visitBlock(UnitCompiler.java:1376) at org.codehaus.janino.UnitCompiler$6.visitBlock(UnitCompiler.java:1370) at org.codehaus.janino.Java$Block.accept(Java.java:2471) at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1370) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:2220) at org.codehaus.janino.UnitCompiler.access$1800(UnitCompiler.java:206) at org.codehaus.janino.UnitCompiler$6.visitIfStatement(UnitCompiler.java:1378) at org.codehaus.janino.UnitCompiler$6.visitIfStatement(UnitCompiler.java:1370) at org.codehaus.janino.Java$IfStatement.accept(Java.java:2621) at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1370) at org.codehaus.janino.UnitCompiler.compileStatements(UnitCompiler.java:1450) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:1436) at org.codehaus.janino.UnitCompiler.access$1600(UnitCompiler.java:206) at org.codehaus.janino.UnitCompiler$6.visitBlock(UnitCompiler.java:1376) at org.codehaus.janino.UnitCompiler$6.visitBlock(UnitCompiler.java:1370) at org.codehaus.janino.Java$Block.accept(Java.java:2471) at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1370) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:2220) at org.codehaus.janino.UnitCompiler.access$1800(UnitCompiler.java:206) at org.codehaus.janino.UnitCompiler$6.visitIfStatement(UnitCompiler.java:1378) at org.codehaus.janino.UnitCompiler$6.visitIfStatement(UnitCompiler.java:1370) at org.codehaus.janino.Java$IfStatement.accept(Java.java:2621) at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1370) at org.codehaus.janino.UnitCompiler.compileStatements(UnitCompiler.java:1450) at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:2811) at org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1262) at org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1234) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:538) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:890) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:894) at org.codehaus.janino.UnitCompiler.access$600(UnitCompiler.java:206) at org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:377) at org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:369) at org.codehaus.janino.Java$MemberClassDeclaration.accept(Java.java:1128) at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:369) at org.codehaus.janino.UnitCompiler.compileDeclaredMemberTypes(UnitCompiler.java:1209) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:564) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:420) at org.codehaus.janino.UnitCompiler.access$400(UnitCompiler.java:206) at org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:374) at org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:369) at org.codehaus.janino.Java$AbstractPackageMemberClassDeclaration.accept(Java.java:1309) at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:369) at org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:345) at org.codehaus.janino.SimpleCompiler.compileToClassLoader(SimpleCompiler.java:396) at org.codehaus.janino.ClassBodyEvaluator.compileToClass(ClassBodyEvaluator.java:311) at org.codehaus.janino.ClassBodyEvaluator.cook(ClassBodyEvaluator.java:229) at org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:196) at org.codehaus.commons.compiler.Cookable.cook(Cookable.java:91) at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$.org$apache$spark$sql$catalyst$expressions$codegen$CodeGenerator$$doCompile(CodeGenerator.scala:960) ... 36 more
https://issues.apache.org/jira/browse/SPARK-16845 refers that it was fixed and applied to 2.2.0, but I'm still observing this on AWS EMR 5.8.0. Can you please help?