Uploaded image for project: 'jclouds'
  1. jclouds
  2. JCLOUDS-767

Creating multiple spot instances fails. Status polling too frequent?

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Blocker
    • Resolution: Unresolved
    • 1.8.1
    • None
    • jclouds-compute
    • AWS, Spot instances, CoreOS

    Description

      My program creates spot instances (see 'addNode' below).
      Recently, in some cases I've been getting "RequestResourceCountExceeded" error (see below).

      I am creating multiple nodes on multiple threads (3).
      It appears that jClouds is polling node status too frequently, and AWS perceives this as a DOS attack. AWS polling queries fail on "The amount of data you requested and frequency of requests together exceed the levels allowed by the Spot service".

      jClouds interprets this as a failure to create, whereas this is just a failure to poll.

      The spot requests, however, are valid - and eventually fulfilled, but because of this error - they are never properly configured by jClouds, and in fact - are just left as orphans.

      This scenario, in turn, ends up creating more and more nodes - endlessly, all left as orphans (but costing dear $).

      So,
      1) This specific error should not be interpreted as failure
      2) The polling interval should be increased, or at least configurable
      3) Maybe some sort of exponential backoff should be used in case this error happens
      4) Failed spot requests should verify that the spot request is cancelled

      (the code below happens on 3 threads, concurrently)
      public ComputeMetadata addNode(Template template) throws RunNodesException {
      ComputeMetadata result = Iterables.getOnlyElement(getComputeService().createNodesInGroup(getGroupName(), 1, template));
      LOG.info("Node \"{}\" created", result.getName());

      10:06:02.795 [main] INFO io.multicloud.controller.AWSProvider - quorum is missing 3 nodes
      10:06:03.077 [main] INFO i.m.controller.CloudProviderImpl - Adding 3 nodes
      10:06:04.113 [pool-3-thread-1] WARN jclouds.compute - << not api visible instances([

      {region=us-east-1, name=sir-02gebgm2}

      ])
      10:06:04.342 [pool-3-thread-3] WARN jclouds.compute - << not api visible instances([

      {region=us-east-1, name=sir-02gez2h5}

      ])
      10:06:04.460 [pool-3-thread-2] WARN jclouds.compute - << not api visible instances([

      {region=us-east-1, name=sir-02ghn9t2}

      ])
      10:06:17.853 [user thread 1] ERROR jclouds.compute - << problem customizing node(us-east-1/sir-02gez2h5):
      org.jclouds.aws.AWSResponseException: request POST https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, error: AWSError{requestId='c5ffce45-02c3-4443-b4c2-42cfe3e2a7ef', requestToken='null', code='RequestResourceCountExceeded', message='the amount of data you requested and frequency of requests together exceed the levels allowed by the Spot service', context='

      {Response=, Errors=}'}
      at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75) ~[sts-1.8.1.jar:1.8.1]
      at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65) ~[jclouds-core-1.8.1.jar:1.8.1]
      at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:128) ~[jclouds-core-1.8.1.jar:1.8.1]
      at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:98) ~[jclouds-core-1.8.1.jar:1.8.1]
      at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90) ~[jclouds-core-1.8.1.jar:1.8.1]
      at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73) ~[jclouds-core-1.8.1.jar:1.8.1]
      at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44) ~[jclouds-core-1.8.1.jar:1.8.1]
      at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117) ~[jclouds-core-1.8.1.jar:1.8.1]
      at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87) ~[guava-17.0.jar:na]
      at $Proxy70.describeSpotInstanceRequestsInRegion(Unknown Source) ~[na:na]
      at org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:53) ~[aws-ec2-1.8.1.jar:1.8.1]
      at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56) ~[ec2-1.8.1.jar:1.8.1]
      at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:58) ~[jclouds-compute-1.8.1.jar:1.8.1]
      at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:38) ~[jclouds-compute-1.8.1.jar:1.8.1]
      at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:57) ~[jclouds-compute-1.8.1.jar:1.8.1]
      at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:40) ~[jclouds-compute-1.8.1.jar:1.8.1]
      at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:142) ~[jclouds-compute-1.8.1.jar:1.8.1]
      at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:137) ~[jclouds-compute-1.8.1.jar:1.8.1]
      at org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111) ~[jclouds-core-1.8.1.jar:1.8.1]
      at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:154) ~[jclouds-compute-1.8.1.jar:1.8.1]
      at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:118) ~[jclouds-compute-1.8.1.jar:1.8.1]
      at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:63) ~[jclouds-compute-1.8.1.jar:1.8.1]
      at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:42) ~[jclouds-compute-1.8.1.jar:1.8.1]
      at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:117) [jclouds-compute-1.8.1.jar:1.8.1]
      at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49) [jclouds-compute-1.8.1.jar:1.8.1]
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [na:1.7.0_09]
      at java.util.concurrent.FutureTask.run(FutureTask.java:166) [na:1.7.0_09]
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_09]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_09]
      at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09]
      10:06:17.872 [pool-3-thread-3] ERROR i.m.controller.CloudProviderImpl - Failed to create node:
      org.jclouds.compute.RunNodesException: error running 1 node group(AWS-East-1) location(us-east-1) image(ami-06ff446e) size(m1.medium) options({userMetadata={TYPE=SPOT}, keyPair=Yaron, userDataCksum=c242210b, subnetId=subnet-4f5a433b, spotPrice=0.013, groupIds=[sg-ce6512ab, sg-d9cb77bc]})
      Execution failures:

      0 error[s]
      Node failures:

      1) AWSResponseException on node us-east-1/sir-02gez2h5:
      org.jclouds.aws.AWSResponseException: request POST https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, error: AWSError{requestId='c5ffce45-02c3-4443-b4c2-42cfe3e2a7ef', requestToken='null', code='RequestResourceCountExceeded', message='the amount of data you requested and frequency of requests together exceed the levels allowed by the Spot service', context='{Response=, Errors=}

      '}
      at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
      at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
      at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:128)
      at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:98)
      at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
      at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
      at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
      at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
      at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
      at $Proxy70.describeSpotInstanceRequestsInRegion(Unknown Source)
      at org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:53)
      at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56)
      at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:58)
      at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:38)
      at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:57)
      at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:40)
      at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:142)
      at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:137)
      at org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111)
      at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:154)
      at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:118)
      at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:63)
      at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:42)
      at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:117)
      at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
      at java.util.concurrent.FutureTask.run(FutureTask.java:166)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
      at java.lang.Thread.run(Thread.java:722)

      1 error[s]
      at org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:222) ~[jclouds-compute-1.8.1.jar:1.8.1]
      at org.jclouds.ec2.compute.EC2ComputeService.createNodesInGroup(EC2ComputeService.java:148) ~[ec2-1.8.1.jar:1.8.1]
      at io.multicloud.controller.CloudProviderImpl.addNode(CloudProviderImpl.java:65) ~[classes/:na]
      at io.multicloud.controller.CloudProviderImpl$CreateNewNodeTask.run(CloudProviderImpl.java:227) ~[classes/:na]
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_09]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_09]
      at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09]
      10:07:47.218 [user thread 0] ERROR jclouds.compute - << problem customizing node(us-east-1/sir-02gebgm2):
      org.jclouds.aws.AWSResponseException: request POST https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, error: AWSError{requestId='82736b80-b101-4111-acd0-69e3ce53d645', requestToken='null', code='RequestResourceCountExceeded', message='the amount of data you requested and frequency of requests together exceed the levels allowed by the Spot service', context='

      {Response=, Errors=}'}
      at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75) ~[sts-1.8.1.jar:1.8.1]
      at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65) ~[jclouds-core-1.8.1.jar:1.8.1]
      at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:128) ~[jclouds-core-1.8.1.jar:1.8.1]
      at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:98) ~[jclouds-core-1.8.1.jar:1.8.1]
      at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90) ~[jclouds-core-1.8.1.jar:1.8.1]
      at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73) ~[jclouds-core-1.8.1.jar:1.8.1]
      at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44) ~[jclouds-core-1.8.1.jar:1.8.1]
      at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117) ~[jclouds-core-1.8.1.jar:1.8.1]
      at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87) ~[guava-17.0.jar:na]
      at $Proxy70.describeSpotInstanceRequestsInRegion(Unknown Source) ~[na:na]
      at org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:53) ~[aws-ec2-1.8.1.jar:1.8.1]
      at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56) ~[ec2-1.8.1.jar:1.8.1]
      at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:58) ~[jclouds-compute-1.8.1.jar:1.8.1]
      at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:38) ~[jclouds-compute-1.8.1.jar:1.8.1]
      at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:57) ~[jclouds-compute-1.8.1.jar:1.8.1]
      at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:40) ~[jclouds-compute-1.8.1.jar:1.8.1]
      at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:142) ~[jclouds-compute-1.8.1.jar:1.8.1]
      at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:137) ~[jclouds-compute-1.8.1.jar:1.8.1]
      at org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111) ~[jclouds-core-1.8.1.jar:1.8.1]
      at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:154) ~[jclouds-compute-1.8.1.jar:1.8.1]
      at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:118) ~[jclouds-compute-1.8.1.jar:1.8.1]
      at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:63) ~[jclouds-compute-1.8.1.jar:1.8.1]
      at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:42) ~[jclouds-compute-1.8.1.jar:1.8.1]
      at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:117) [jclouds-compute-1.8.1.jar:1.8.1]
      at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49) [jclouds-compute-1.8.1.jar:1.8.1]
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [na:1.7.0_09]
      at java.util.concurrent.FutureTask.run(FutureTask.java:166) [na:1.7.0_09]
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_09]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_09]
      at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09]
      10:07:47.220 [pool-3-thread-1] ERROR i.m.controller.CloudProviderImpl - Failed to create node:
      org.jclouds.compute.RunNodesException: error running 1 node group(AWS-East-1) location(us-east-1) image(ami-06ff446e) size(m1.medium) options({userMetadata={TYPE=SPOT}, keyPair=Yaron, userDataCksum=c242210b, subnetId=subnet-0afda622, spotPrice=0.013, groupIds=[sg-ce6512ab, sg-d9cb77bc]})
      Execution failures:

      0 error[s]
      Node failures:

      1) AWSResponseException on node us-east-1/sir-02gebgm2:
      org.jclouds.aws.AWSResponseException: request POST https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, error: AWSError{requestId='82736b80-b101-4111-acd0-69e3ce53d645', requestToken='null', code='RequestResourceCountExceeded', message='the amount of data you requested and frequency of requests together exceed the levels allowed by the Spot service', context='{Response=, Errors=}

      '}
      at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
      at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
      at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:128)
      at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:98)
      at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
      at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
      at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
      at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
      at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
      at $Proxy70.describeSpotInstanceRequestsInRegion(Unknown Source)
      at org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:53)
      at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56)
      at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:58)
      at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:38)
      at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:57)
      at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:40)
      at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:142)
      at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:137)
      at org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111)
      at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:154)
      at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:118)
      at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:63)
      at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:42)
      at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:117)
      at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
      at java.util.concurrent.FutureTask.run(FutureTask.java:166)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
      at java.lang.Thread.run(Thread.java:722)

      1 error[s]
      at org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:222) ~[jclouds-compute-1.8.1.jar:1.8.1]
      at org.jclouds.ec2.compute.EC2ComputeService.createNodesInGroup(EC2ComputeService.java:148) ~[ec2-1.8.1.jar:1.8.1]
      at io.multicloud.controller.CloudProviderImpl.addNode(CloudProviderImpl.java:65) ~[classes/:na]
      at io.multicloud.controller.CloudProviderImpl$CreateNewNodeTask.run(CloudProviderImpl.java:227) ~[classes/:na]
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_09]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_09]
      at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09]
      10:09:24.051 [pool-3-thread-2] INFO i.m.controller.CloudProviderImpl - Node "AWS-East-1-8b5f5f66" created
      10:09:24.051 [pool-3-thread-2] INFO i.m.controller.CloudProviderImpl - Adding node "AWS-East-1-8b5f5f66" to load balancer
      10:09:25.110 [pool-3-thread-2] INFO i.m.controller.CloudProviderImpl - Creating node using template: {image={id=us-east-1/ami-06ff446e, providerId=ami-06ff446e, name=CoreOS-beta-444.5.0, location=

      {scope=REGION, id=us-east-1, description=us-east-1, parent=aws-ec2, iso3166Codes=[US-VA]}, os={family=unrecognized, arch=paravirtual, version=, description=595879546273/CoreOS-beta-444.5.0, is64Bit=true}, description=CoreOS beta 444.5.0 (PV), status=AVAILABLE[available], loginUser=root, userMetadata={owner=595879546273, rootDeviceType=ebs, virtualizationType=paravirtual, hypervisor=xen}}, hardware={id=m1.medium, providerId=m1.medium, processors=[{cores=1.0, speed=2.0}], ram=3750, volumes=[{type=LOCAL, size=10.0, device=/dev/sda1, bootDevice=true, durable=false}, {type=LOCAL, size=420.0, device=/dev/sdb, bootDevice=false, durable=false}, {type=LOCAL, size=420.0, device=/dev/sdc, bootDevice=false, durable=false}], supportsImage=Predicates.and(Predicates.alwaysTrue(),Predicates.or(isWindows(),requiresVirtualizationType(paravirtual)),Predicates.alwaysTrue(),Predicates.alwaysTrue())}, location={scope=REGION, id=us-east-1, description=us-east-1, parent=aws-ec2, iso3166Codes=[US-VA]}

      , options={userMetadata=

      {TYPE=SPOT}

      , keyPair=Yaron, userDataCksum=c242210b, subnetId=subnet-50cdf616, spotPrice=0.013, groupIds=[sg-ce6512ab, sg-d9cb77bc]}}

      Attachments

        Activity

          People

            Unassigned Unassigned
            yaronr Yaron Rosenbaum
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: