Avoid unexpected charges on your AWS bill by automating cost monitoring and reporting using AWS Lambda and Boto3.

In this article, we will explore how to use Python’s Boto3 library and various AWS services to effectively manage and track your AWS costs.

As an avid AWS user, you may have encountered unexpected costs on your monthly bill. This can be a common occurrence, particularly when using the AWS Management Console for your work. Monitoring and managing your AWS resources can be challenging, as it’s easy to lose track of them, especially if you are regularly creating and deleting resources for practices or demonstrations. Neglecting to delete unused resources can lead to unexpected and potentially costly charges on your AWS bill.

To avoid unexpected costs, there are some AWS Tools which we can use. For example, we have Budgets where we can get notified(via email) if the bill exceeds the limit.

Here is an example of an email alert.

From the above email, we can infer that the current billed amount is $1.71 and we got this email as we have crossed the alert threshold of $1.60.

But as you can see, we don't get any data on which resources are getting billed. To know the cost incurred by resources, we can use AWS Cost Explorer.

AWS Cost Explorer

It is a tool that we can use to visualize and manage AWS Costs and Usage. This tool provides us with graphs, reports, alerts and forecasts. We can access AWS Cost Explorer via AWS Management Console. This is a free-to-use tool.

Here is an example cost and usage report from cost explorer.

By setting the Dimension attribute as Service, you were able to view the costs incurred by all AWS services within the specified date range of September 1, 2022 to November 30, 2022. Upon reviewing the data, you noticed that in the month of October, you had created some VPC Endpoints that were not subsequently deleted. This resulted in a daily cost of approximately $6, as reflected in the graph under the VPC resource. It’s important to regularly monitor your AWS resources and costs to ensure that you are not incurring unnecessary expenses. In this case, simply deleting the unused VPC Endpoints would have prevented this unexpected cost.

Proposed Solution

Here, we will use a Python Boto3 function that leverages the AWS Cost Explorer API to retrieve cost and usage data. We will then convert the function into a Lambda function which sends us an email with obtained data as a body using Amazon SES and schedule it to run on a regular basis using Amazon EventBridge.

Note: The AWS Cost Explorer API lets you directly access the interactive, ad-hoc query engine that powers AWS Cost Explorer. Each request will incur a cost of $0.01

If we run once a week, we incur 0.04 dollars as API cost which is negligible.

Python Function

To get an up-to-date view of your current AWS bill, run this function. The bill period is set to the current month up to the present day. To use this function, you must have AWS CLI configured on your system and have the Boto3 library installed (use pip install boto3).

The get_cost_and_usage() function in AWS Cost Explorer allows you to retrieve cost and usage data for your AWS resources. In this function, you can specify the time period you want to retrieve data for, as well as the metrics and dimensions you want to include in the report.

In this case, I have specified a period that begins at the start of the current month and extends to the present day. I have also specified the “UnblendedCost” metric, which represents the raw cost of a resource before any discounts or credits are applied. If you want to include the effect of discounts and credits in your report, you can use the “BlendedCost” metric instead.

I included a filter to exclude any credits or refunds from the report. Finally, I have specified the “Service” dimension, which will group the data in your report by AWS service. By combining these different parameters, I create a customized report that provides valuable insights into your AWS costs and resource usage.

The console output of the function would be something like the below pic.

Converting it into a Lambda Function

In this case, you are creating an AWS Lambda function that utilizes the Cost Explorer API and Amazon Simple Email Service (SES) to retrieve and email your AWS cost data. The Lambda function will be written in Python 3.9 and will need the appropriate permissions to access the Cost Explorer API and SES. It is important to carefully select an IAM role with the necessary permissions, rather than using the highly-permissive “AdministratorAccess” policy.

Before you can use SES to send emails, you will need to verify the email address that you want to use. This ensures that you have permission to send emails from that address and helps to protect the integrity of the service. To verify an email address, simply visit the Amazon SES dashboard and follow the prompts to complete the verification process.

By creating this Lambda function and utilizing the Cost Explorer API and SES, you can automate the process of tracking and reporting your AWS costs, saving time and effort while ensuring that you have the latest data at your fingertips.

Lambda Function Code:-

The output of the email will be as follows:-

Running the function Periodically

To automate the execution of your AWS Lambda function on a regular basis, you can use the Amazon EventBridge service to create a scheduled event. EventBridge supports two types of schedules: cron-based and rate-based. The cron-based schedule offers more flexibility, but the rate-based schedule is simpler to set up and may be more suitable for certain use cases.

In this case, I have chosen to use a rate-based schedule to trigger Lambda function. With this type of schedule, I can specify the frequency at which the event should occur, such as once per day or once per week. This allows you to easily run your function on a predetermined schedule, without the need to manually invoke it each time.

The scheduled pattern which I used is shown below.

To complete the process of setting up your scheduled Lambda function, you will need to specify the target for the event. In this case, you will select the “Lambda” API as the target and then choose your Lambda function from the dropdown menu. This will associate your function with the scheduled event, so that it will be executed according to the schedule you have configured.

Once you have selected the target and chosen your Lambda function, you can proceed to create the schedule. This will activate the event and your function will begin executing according to the schedule you have specified. By using EventBridge and a scheduled event, you can automate the execution of your Lambda function and ensure that your AWS cost and usage data is regularly retrieved and processed.

To wrap up, it is important to regularly monitor and manage your AWS resources and costs to avoid unexpected expenses and optimize your use of the platform. By using tools like the AWS Cost Explorer, cost allocation tags, and the AWS Budgets service, you can gain a better understanding of your resource usage and costs and identify opportunities to reduce expenses.