Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-6016

Tensor flow Chicago taxi failing for 256MB data

Details

    Description

      Chicago taxi example is failing when runningĀ on 1 node cluster with 256MB data.

      Exception:

      018-11-07 15:20:10,736 INFO org.apache.flink.runtime.taskmanager.Task - GroupReduce (GroupReduce at Analyze/ComputeAnalyzerOutputs[0]/Analyze[scale_to_z_score_1/mean_and_var/mean/sum/]/CombineGlobally(_CombineFnWrapper)/CombinePerKey/GroupByKey) (1/1) (6be52aa98e1f7233172f58eb8695fb6d) switched
      from RUNNING to FAILED.
      java.lang.Exception: The data preparation for task 'GroupReduce (GroupReduce at Analyze/ComputeAnalyzerOutputs[0]/Analyze[scale_to_z_score_1/mean_and_var/mean/sum/]/CombineGlobally(_CombineFnWrapper)/CombinePerKey/GroupByKey)' , caused an error: Error obtaining the sorted input: Thread 'SortMerger Reading Thread' ter
      minated due to an exception: The record exceeds the maximum size of a sort buffer (current maximum: 4456448 bytes).
       at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:479)
       at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:368)
       at org.apache.flink.runtime.taskmanager.Task.run(Task.java:712)
       at java.lang.Thread.run(Thread.java:748)
      Caused by: java.lang.RuntimeException: Error obtaining the sorted input: Thread 'SortMerger Reading Thread' terminated due to an exception: The record exceeds the maximum size of a sort buffer (current maximum: 4456448 bytes).
       at org.apache.flink.runtime.operators.sort.UnilateralSortMerger.getIterator(UnilateralSortMerger.java:650)
       at org.apache.flink.runtime.operators.BatchTask.getInput(BatchTask.java:1108)
       at org.apache.flink.runtime.operators.GroupReduceDriver.prepare(GroupReduceDriver.java:99)
       at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:473)
       ... 3 more
      Caused by: java.io.IOException: Thread 'SortMerger Reading Thread' terminated due to an exception: The record exceeds the maximum size of a sort buffer (current maximum: 4456448 bytes).
       at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:831)
      Caused by: java.io.IOException: The record exceeds the maximum size of a sort buffer (current maximum: 4456448 bytes).
       at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ReadingThread.go(UnilateralSortMerger.java:986)
       at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:827)
      2018-11-07 15:20:10,741 ERROR org.apache.flink.runtime.operators.BatchTask - Error in task code: GroupReduce (GroupReduce at Analyze/ComputeAnalyzerOutputs[0]/Analyze[scale_to_z_score/mean_and_var/mean/sum/]/CombineGlobally(_CombineFnWrapper)/CombinePerKey/GroupByKey) (1/1)
      java.lang.Exception: The data preparation for task 'GroupReduce (GroupReduce at Analyze/ComputeAnalyzerOutputs[0]/Analyze[scale_to_z_score/mean_and_var/mean/sum/]/CombineGlobally(_CombineFnWrapper)/CombinePerKey/GroupByKey)' , caused an error: Error obtaining the sorted input: Thread 'SortMerger Reading Thread' termi
      nated due to an exception: The record exceeds the maximum size of a sort buffer (current maximum: 4456448 bytes).
       at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:479)
       at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:368)
       at org.apache.flink.runtime.taskmanager.Task.run(Task.java:712)
       at java.lang.Thread.run(Thread.java:748)
      Caused by: java.lang.RuntimeException: Error obtaining the sorted input: Thread 'SortMerger Reading Thread' terminated due to an exception: The record exceeds the maximum size of a sort buffer (current maximum: 4456448 bytes).
       at org.apache.flink.runtime.operators.sort.UnilateralSortMerger.getIterator(UnilateralSortMerger.java:650)
       at org.apache.flink.runtime.operators.BatchTask.getInput(BatchTask.java:1108)
       at org.apache.flink.runtime.operators.GroupReduceDriver.prepare(GroupReduceDriver.java:99)
       at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:473)
       ... 3 more
      Caused by: java.io.IOException: Thread 'SortMerger Reading Thread' terminated due to an exception: The record exceeds the maximum size of a sort buffer (current maximum: 4456448 bytes).
       at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:831)
      Caused by: java.io.IOException: The record exceeds the maximum size of a sort buffer (current maximum: 4456448 bytes).
       at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ReadingThread.go(UnilateralSortMerger.java:986)
       at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:827)
      2018-11-07 15:20:10,743 INFO org.apache.flink.runtime.taskmanager.Task - GroupReduce (GroupReduce at Analyze/ComputeAnalyzerOutputs[0]/Analyze[scale_to_z_score/mean_and_var/mean/sum/]/CombineGlobally(_CombineFnWrapper)/CombinePerKey/GroupByKey) (1/1) (ae0b703b1e3a194644b32a2fa534436f) switched fr
      om RUNNING to FAILED.
      java.lang.Exception: The data preparation for task 'GroupReduce (GroupReduce at Analyze/ComputeAnalyzerOutputs[0]/Analyze[scale_to_z_score/mean_and_var/mean/sum/]/CombineGlobally(_CombineFnWrapper)/CombinePerKey/GroupByKey)' , caused an error: Error obtaining the sorted input: Thread 'SortMerger Reading Thread' termi
      nated due to an exception: The record exceeds the maximum size of a sort buffer (current maximum: 4456448 bytes).
       at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:479)
       at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:368)
       at org.apache.flink.runtime.taskmanager.Task.run(Task.java:712)
       at java.lang.Thread.run(Thread.java:748)
      Caused by: java.lang.RuntimeException: Error obtaining the sorted input: Thread 'SortMerger Reading Thread' terminated due to an exception: The record exceeds the maximum size of a sort buffer (current maximum: 4456448 bytes).
       at org.apache.flink.runtime.operators.sort.UnilateralSortMerger.getIterator(UnilateralSortMerger.java:650)
       at org.apache.flink.runtime.operators.BatchTask.getInput(BatchTask.java:1108)
       at org.apache.flink.runtime.operators.GroupReduceDriver.prepare(GroupReduceDriver.java:99)
       at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:473)
       ... 3 more
      Caused by: java.io.IOException: Thread 'SortMerger Reading Thread' terminated due to an exception: The record exceeds the maximum size of a sort buffer (current maximum: 4456448 bytes).
       at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:831)
      Caused by: java.io.IOException: The record exceeds the maximum size of a sort buffer (current maximum: 4456448 bytes).
       at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ReadingThread.go(UnilateralSortMerger.java:986)
       at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:827)
      
      

      Attachments

        Issue Links

          Activity

            People

              angoenka Ankur Goenka
              angoenka Ankur Goenka
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: