Details
-
Bug
-
Status: Resolved
-
P3
-
Resolution: Not A Problem
-
2.9.0
-
None
-
MacOS
Description
Maybe I've done something wrong, but when you try to access a field this has been generated in a SqlTransform it throws an exception:
java.lang.RuntimeException: org.apache.beam.sdk.util.UserCodeException: java.lang.IllegalArgumentException: Cannot find field views in schema Fields: Field{name=wikimedia_project, description=, type=FieldType{typeName=STRING, collectionElementType=null, collectionElementTypeNullable=null, mapKeyType=null, mapValueType=null, mapValueTypeNullable=null, rowSchema=null, metadata=null}, nullable=false} Field{name=EXPR$1, description=, type=FieldType{typeName=INT32, collectionElementType=null, collectionElementTypeNullable=null, mapKeyType=null, mapValueType=null, mapValueTypeNullable=null, rowSchema=null, metadata=null}, nullable=false}
Instead of being able to access the `views` field, it has been named `EXPR$1` by Beam/Dataflow. So, to get the value of the field I need to do this:
bqRow.set("views", row.getInt32("EXPR$1"));
instead of:
bqRow.set("views", row.getInt32("views"));
PCollection<Row> outputStream = sqlRows.setRowSchema(SCHEMA) .apply("sql_transform", SqlTransform.query( "select wikimedia_project, sum(views) " + "from PCOLLECTION " + "group by wikimedia_project"));
Pipeline is reading a file from GCS, transforming it (using SqlTransform) and writing to BigQuery. Code can be found here: