Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-12867

Either Create or DirectRunner fails to produce all elements to the following transform

Details

    • Bug
    • Status: Open
    • P1
    • Resolution: Unresolved
    • None
    • None
    • runner-direct
    • None

    Description

      The following pipeline fails to print out all numbers 1 to 100 when run on DirectRunner in streaming mode.

      This was identified implementing org.apache.beam.sdk.io.gcp.pubsublite.ReadWriteIT, which uses a workaround for the bug in either Create or DirectRunner:

      private static final int COUNT = 100;
      
      private static AtomicInteger CREATED_COUNT = new AtomicInteger();
      
      public static void run(Pipeline pipeline) {
        PCollection<Integer> indexes = pipeline.apply( "createIndexes",
            Create.of(IntStream.range(0, COUNT).boxed().collect(Collectors.toList())));
      
      indexes.apply(
       "createMessages",
       MapElements.via(
           new SimpleFunction<Integer, Integer>(
               index -> {
                 System.err.println("Created message index " + createdCount.incrementAndGet());
                 return index;
               }) {}));
      
        pipeline.run().waitUntilFinish();  // Never terminates
      }
      

      Attachments

        Activity

          People

            Unassigned Unassigned
            dpcollins-google Daniel Collins
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated: