Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-190

the allocation of workers, executors and tasks is amazing in our storm-0.9.0.1 cluster

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Not A Problem
    • None
    • None
    • storm-core
    • storm-0.9.0.1

    Description

      we are now testing storm-0.9.0.1 cluster for migrating. When running the topologies in storm-starter, such as WordCountTopology, ReachTopology, we found the number of workers, executors, and tasks is unexplainable (according to https://github.com/nathanmarz/storm/wiki/Understanding-the-parallelism-of-a-Storm-topology ):

      1) WordCountTopology:
      Name Id Status Uptime Num workers Num executors Num tasks
      wc-test3 wc-test3-3-1388986658 ACTIVE 29m 5s 25 50 50
      wc-test2 wc-test2-2-1388986111 ACTIVE 38m 12s 4 29 29
      wc-test wc-test-1-1388981297 ACTIVE 1h 58m 26s 3 28 28

      This topology's configuration is as follows:
      TopologyBuilder builder = new TopologyBuilder();
      builder.setSpout("spout", new RandomSentenceSpout(), 5);
      builder.setBolt("split", new SplitSentence(), 8).shuffleGrouping("spout");
      builder.setBolt("count", new WordCount(), 12).fieldsGrouping("split", new Fields("word"));
      ......
      conf.setNumWorkers(3); #4, 25

      2) ReachTopology (which is a drpc topology):
      Name Id Status Uptime Num workers Num executors Num tasks
      reach-test reach-test-4-1388986889 ACTIVE 32m 20s 6 35 35

      This topology's configuration is as follows:
      LinearDRPCTopologyBuilder builder = new LinearDRPCTopologyBuilder("reach");
      builder.addBolt(new GetTweeters(), 4);
      builder.addBolt(new GetFollowers(), 12).shuffleGrouping();
      builder.addBolt(new PartialUniquer(), 6).fieldsGrouping(new Fields("id", "follower"));
      builder.addBolt(new CountAggregator(), 3).fieldsGrouping(new Fields("id"));
      ......
      conf.setNumWorkers(6);

      Attachments

        Activity

          People

            Unassigned Unassigned
            panfeng412 Panfeng Yuan
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - 36h
                36h
                Remaining:
                Remaining Estimate - 36h
                36h
                Logged:
                Time Spent - Not Specified
                Not Specified