Nemo on AWS Lambda

AWS Lambda (and other cloud serverless frameworks) is increasingly widely used to run data analytics computations. So it'd be nice to enable Nemo to run on AWS Lambda, especially as one of the strengths of Nemo is the ease of enforcing resource-aware scheduling.

Although Nemo at the moment does support different types of resources such as transient and geo-distributed resources, these resources are assumed to be managed by a central resource manager and generally consumed by a Java process that starts with the standard main() without particular restriction.

However, these assumptions do not hold for the resources on AWS Lambda.

• Resources on AWS Lambda can only be used through deploying a function using a particular programming model and API enforced by AWS
• Functions running on AWS Lambda cannot communicate via network with each other directly
• Functions generally cannot share memory (e.g., shared variable)

Hence, a simple optimization pass such as TransientResourcePass or GeoDistPass would not suffice for Nemo to harness resources on AWS Lambda.

The following is a list of components/features that needs to be introduced to Nemo. (the list can grow of course as we develop and discover new things)

• Mechanisms to start/stop AWS Lambda
• AWS credentials management
• A new Nemo 'Executor' for AWS Lambda to run Nemo 'Task's
• Data (re)routing mechanisms for transferring data back and forth with AWS Lambda
• Scheduling mechanisms to (perhaps dynamically) decide which subset of computations to run on AWS Lambda