Automating Little Things With Lambda
When you need to automate a little thing in RONIN, look towards AWS Lambda.
Sometimes you just want a little bit of compute to do something for you. System administrators often use the
cron utility to remember to run things on a schedule. However, the machine that runs
cron must be running when you have scheduled events, which can end up costing more than you really need for a few occasional tasks that take a few seconds. Enter AWS Lambda, a service that allows you to run a wee bit of code in response to a trigger (such as something happening in your AWS account, or a "timer" you have set using CloudWatch). You don't need to manage a machine under the hood. Little bits of code together are incredibly powerful!
As a RONIN administrator with AWS Console access, there are some functions you may want to automate. For example, we posted about a way to generate detailed billing reports. You may want to create a Lambda function to generate a report and send it to yourself each week. Or perhaps you want to automate some other function, such as making sure that every file written to a collaborative research project's object store conforms to a particular naming convention (and if not, notifying the author and moving it out)!
This blog will walk you through the steps involved in creating a Lambda function and triggering it on a schedule. We assume that you feel comfortable writing the actual code of the Lambda function, or have a sample of what you want to do.
All of these steps will need to be completed from the AWS Console.
Step 1. Create a Lambda Function
Navigate to the Lambda service page (you can type in Lambda in the top search bar). Select Create Function (Figure 1).
Select Author from scratch.
Set the function name to be anything descriptive.
Change the runtime if you wish to be the language in which you intend to write your function.
Change the architecture if you wish to run on a different architecture. If you need to do this you will probably know.
Allow Lambda to create an execution role to upload logs to Amazon CloudWatch Logs. This role describes the permissions that your function has when it executes (e.g. what AWS resources it can access). You can edit this role later to add permissions for the Lambda function to send mail, write data to an S3 bucket of your choosing, or do anything else you need it to.
Click on "Create Function" on the lower right.
Step 2. Edit the Lambda Function
By default, creating a function from scratch will set you up with a little "Hello World". The handler function processes events when they occur; you can read more about the arguments to this function. This particular "Hello World" function doesn't need access to any specific AWS resources, so we can just go ahead and test it with a test event that we create using the tab just to the right of "Code". Recall that Lambdas are triggered by events that have information in them that the Lambda function can use. In this case, our Hello World function doesn't use any inputs, so you don't need to edit the default test event either. When you test your Lambda, you should see that the test succeeded, and the output will be in the execution log.
Now is where you will probably want to edit your function to actually do something. For example, you can write code to send an email or maybe you want to preprocess external data on S3 before making it available on a RONIN object store.
Step 3. Edit Lambda Permissions
When you have decided what your Lambda function will do, you need to give it permissions. When doing this, we employ the "principle of least privilege", which means giving the function only those permissions it needs and no more.
Click on Configuration and then Permissions on the left hand side (Figure 3). Here you can access the Execution Role that Lambda created for you. Click on this Execution Role, and you will be catapulted into the IAM service, under Roles. You will see that Lambda has created a Policy on your behalf, which is attached to the Execution Role. You will need to add permissions to the Execution Role to allow it to access things on your behalf. You can do this by either editing the attached Policy, or by creating a new Policy and attaching it to the Execution Role.
Step 4. Triggering Lambda From a Schedule
Many actions within AWS can trigger a Lambda function automatically, but sometimes you might want to invoke your function according to a schedule. To do this within AWS you can use another service called Amazon EventBridge (an extension of Amazon CloudWatch Events)
Navigate to the Amazon EventBridge page, and click on Rules (Figure 4). You can create a rule that triggers regularly, either at a fixed rate or using a
cron expression (I like this site for helping to write cron expressions). At the bottom of the page, you will need to select the Lambda function that you will trigger from this rule.
Whenever you can't find something, make sure you are looking in the right region.
To see whether everything is working as it should, you can go back to your Lambda function and examine the "Monitor" tab (Figure 5). Here you can see various metrics for your Lambda function, and also examine the logs in CloudWatch.
Last Step: Things To Watch For
As you start working on more and more complex functions you might find that your Lambda functions fail. Unlike running a job using
cron, where a job has access to the resources of the server, a Lambda function has bounded compute resources (memory, execution time). If you get an error, check it carefully to see if you are running out of memory or time, and then change the settings.
Another way that your Lambda might break is if you extend it to access a new AWS service for which it does not have permissions. To fix this, you will need to edit the Lambda permissions (go back to Step 3).
As you move on from simple tasks to more complicated functions that depend upon other packages, you may need to create a deployment package with all these dependencies. We believe this marks a transition towards cloud nativity!