Automating RONIN Cost Reports with AWS Cost Explorer

RONIN's built-in budget management tools for RONIN Administrators and for Project Administrators are excellent for getting a snapshot of current spend, and a breakdown of the major components of that cost. However, often RONIN users and administrators want to keep a closer watchful eye on where that money is going, to answer questions such as:

• How has my spend been changing over the last 6 months?
• What instance types are being used the most?
• Which projects and users are using RONIN the most?
• My spend suddenly increased this last week and who is responsible?

If you have access to the AWS console in the account in which RONIN is installed, or if you know someone who does and can persuade them to generate reports for you, you can answer these questions and more! See our previous blog post for the basics of how to use AWS Cost Explorer to interactively explore your spending.

This blog post will take RONIN administrators to the next level, and explain how to use some really cool open source reporting software to automatically generate and mail you, your researchers, and anyone else incredibly spiffy customizable Excel reports with fun graphs.

Overview of AWS Cost Explorer

Log in to the AWS account in which RONIN is installed so that you can see the console. If you have never used AWS Cost Explorer before, you will need to enable it. If this is the first time you are using AWS Cost Explorer, check out our introductory blog, or continue reading here for a quick refresher.

Because RONIN systematically tags all the resources that are created,  you can break costs down by RONIN Project, billing codes, or even creator.

If you first filter on a project, billing code, or creator, further useful ways to group or filter costs are by Usage Type, or by Usage Type Group. Usage Types are units used to measure usage of different types. They are grouped together into Usage Type Groups. Importantly, you can see things like data transfer costs, EBS snapshot costs, and even API request costs in this category. These costs typically make up a very small percentage of RONIN charges compared to EC2 and storage charges, but you can look at them here.

There is literally no end to the fun you can have by doing this. Below is a link to a very comprehensive tech talk on many aspects of monitoring AWS costs - but if you skip to minute 35, Arthur dives into a nice AWS Cost Explorer demo and highlights how to use it to filter costs over time by service, tags, and to dive deep to answer a variety of questions about spend.

Automating Lovely Reports

At some point you will probably want to generate reports automatically and send them to the people who need them - for example, a RONIN project administrator who doesn't have access to the AWS console might want to see a more detailed breakdown of their project spend or activity by user.

AWS has an application programming interface (API) to automate generation of the data that generates those beautiful pictures in the AWS Cost Explorer tool. However, you need to do the handiwork yourself to actually graph everything. So, we were excited to find an open source sample that you can leverage to automate generation of Excel reports for monthly spend, using the Pandas Excel library to create graphs.

You can run this script from the command line if you have correctly configured AWS credentials for the RONIN AWS account and have verified your email in Amazon SES so that you can send emails (see Prerequisites in the GitHub README). For example, you could create a cron job with a script to generate, save and send reports. Alternatively, you can follow the instructions in the sample to set up a Lambda with an automated AWS Cloudwatch trigger to rig this all in an AWS cloud-native way. In both cases, to run this script, you must have credentials that give you (at a minimum) permissions to send email, write reports to the appropriate S3 bucket, and to access Cost Explorer. You can see the role that is created for Lambda in the deployment YAML file for the sample.

The script allows you to customize the report by editing the main_handler function to include the reports you desire. Each of these reports is a new worksheet in an Excel workbook.

To narrow the reports to a particular project, set the environment variables TAG_VALUE_FILTER and TAG_KEY as follows. For example we can limit reports to the NEURO project as in Figure 3.

export TAG_KEY="RPID"
#change NEURO below to be the value of RPID you wish to select
export TAG_VALUE_FILTER="NEURO"

Below are some examples of reports that you can generate as part of a comprehensive Excel workbook.

#Total cost
costexplorer.addReport(Name="Total", GroupBy=[],Style='Total',IncSupport=True)

#Group by instance type
costexplorer.addReport(Name="InstanceTypes", GroupBy=[{"Type":"DIMENSION", "Key": "INSTANCE_TYPE"}],Style='Total')

#Group by usage type
costexplorer.addReport(Name="Usage Type", GroupBy=[{"Type":"DIMENSION", "Key": "USAGE_TYPE"}],Style='Total')

#Group by services, including support costs
costexplorer.addReport(Name="Services", GroupBy=[{"Type": "DIMENSION","Key": "SERVICE"}],Style='Total',IncSupport=True)

#Group by service, but report on month to month change
costexplorer.addReport(Name="ServicesChange", GroupBy=[{"Type": "DIMENSION","Key": "SERVICE"}],Style='Change')

Finally, to send a report to someone, set the environment variables SES_SEND to the recipient, SES_FROM to be your email address, and SES_REGION to be the region where the SES_FROM address is verified.

Conclusion

Beautiful data on AWS cost is easy to obtain with RONIN tags and AWS Cost Explorer. We recommend thinking about what information your RONIN administrators and users need and setting up an automated method for keeping an eye on spend. We hope this gives you some good ideas for how to do that!

via GIPHY