Kill your AWS Resources when you are done — “the sequel”

More advice on not getting charged for unneeded resources

I have recently written a blog post about a sad situation where I received an unexpected bill from AWS. That was because I made the mistake to follow three different tutorials on EKS deployment with Terraform which resulted in the creation of EC2 instances in 3(!) different regions. That was mainly my fault as I did not pay attention to the autoscaling groups setup and everything kept starting up again and again, even after I was terminating the instances and the cluster itself.

Kill your AWS resources when you are done

I have decided to write a follow-up because of the amazing feedback I received from many people on different Slack channels and Social media. These people had or heard of similar experiences and they resolved the issue by being proactive and disabling the auto-scaling groups on time. Others received an even bigger bill! For example, the following tweet was brought to my attention:

Another person said:

Yes, it’s not a straight forward process setting all these alarms up for billing info.

Another software engineer mentioned that he had the same problem some years ago when the AWS user interface did not show you that you got charged for different regions and you had to change regions in the upper right corner to view the region’s billing 😐. The good news is that AWS has now changed the Billing dashboards and contains multi-region information.

So, I’ve done some further research on the topic. Below I am listing the several options to avoid these unneeded costs. Some of them were pointed out to me by various people and I would like to thank them for that!

Avoid unneeded costs by following these options

Create a budget on AWS

This is a very straightforward way for ensuring you stay within your daily/monthly/quarterly budget. I have configured mine as shown below and I was happy to see that I could extend my setup to trigger an action if I exceed the limit e.g. stop EC2 instance.

AWS Budgets - Amazon Web Services

I took a screenshot of how my AWS budget looks like right now.

my AWS budget dashboard

Tool: Cloud Custodian

I am not going to talk about Cloud custodian, as there are already several blog posts and tutorials on this. My only comment is that apparently, I have used it indirectly in the past in a production environment. The infrastructure team said they managed to reduce the overall AWS monthly billing from the first month! From a developer point of view, I remember that my Fargate cluster was down in the morning and this is how my team learned to be more organised and concerned about costings.

this is how my (development) team learnt to be more organised and concerned about costings

Cloud Custodian official documentation:

Deployment - Cloud Custodian documentation

A use case for Cloud Custodian:

Cloud Custodian — Overview and deployment of cloud governance

Tool: Infracost — Cloud cost estimates for Terraform in pull requests

That sounds amazing! “Infracost is an open-source tool that helps DevOps, SRE, and developers continuously reduce their cloud costs.”

From https://www.infracost.io/

Infracost is a tool I am looking forward to exploring a little more, as it seems a great tool to help developers with costings as well as raise awareness of billing during the development process, which is something that is not always the case. The Github profile of this open-source project looks very attractive and has already collected 3000 stars!

infracost/infracost

Subaccounts with specific permissions

One of the suggestions mentioned the use of different subaccounts, where you can lock regions you don’t want! AWS already does this by default for various regions that are not relevant to you. For example, Tokyo and India's regions are not enabled in my account. If I switch to those regions, a pop-up message will ask me if I really want to enable my access to Tokyo provided that my closest regions are eu-west-1 and eu-west-2.

IAM user/role permissions

The same person (thanks Nico) said that you can also have explicit deny policies in your IAM user/role permissions preventing the same thing using conditions. He also provided the following terraform example that blocks everything outside eu-west-1 (except AWS cert manager in us-east-1 for cloudfront certs).

data "aws_iam_policy_document" "block_aws_regions" {  statement {
effect = "Deny"
not_actions = [
"acm:*",
] resources = ["*"]
condition {
test = "StringNotEquals"
variable = "aws:RequestedRegion"
values = [
"eu-west-1",
"us-east-1",
]
}
} statement {
effect = "Deny"
not_actions = [
"a4b:*",
"aws-marketplace-management:*",
"aws-marketplace:*",
"aws-portal:*",
"awsbillingconsole:*",
"budgets:*",
"ce:*",
"chime:*",
"cloudfront:*",
"config:*",
"cur:*",
"directconnect:*",
"ec2:DescribeRegions",
"ec2:DescribeTransitGateways",
"ec2:DescribeVpnGateways",
"fms:*",
"globalaccelerator:*",
"health:*",
"iam:*",
"importexport:*",
"kms:*",
"mobileanalytics:*",
"networkmanager:*",
"organizations:*",
"pricing:*",
"route53:*",
"route53domains:*",
"s3:GetAccountPublic*",
"s3:ListAllMyBuckets",
"s3:PutAccountPublic*",
"shield:*",
"sts:*",
"support:*",
"trustedadvisor:*",
"waf-regional:*",
"waf:*",
"wafv2:*",
"wellarchitected:*"
] resources = ["*"]
condition {
test = "StringNotEquals"
variable = "aws:RequestedRegion"
values = [
"eu-west-1"
]
}
}

The rise of FinOps

All the above are of course topics covered by FinOps, which is the cultural mindset that aims to save cloud costs for organisations.

FinOps is shorthand for Cloud Financial Management. It is the practice of bringing financial accountability to the variable spend model of cloud, enabling distributed teams to make business trade-offs between speed, cost, and quality. At its core, FinOps is a cultural practice. — https://www.finops.org/what-is-finops/

Similar to the well-known DevOps lifecycle, the FinOps lifecycle aims to iteratively achieve the goal of cost culture and governance.

FinOps lifecycle — operate, inform, optimise

My personal takeaway from my recent research on saving costs is that I would like to explore the world of FinOps. If a random individual received an unexpected bill due to negligence (hello, that’s me 🙂), then it must be a lot harder and a lot more expensive for big organisations, where the Cloud Billing outcome is subject to the actions of several individuals across the teams.

Thank you!

Before you go away, I would like to thank you for reaching at this point. I really hope I have managed to transfer some knowledge to you and the people around you. Yes, that’s the plan! Sharing knowledge is my passion and I am looking forward to keeping doing it. Who knows? One day I may be able to publish a book on Cloud Infrastructure and relevant topics :)

If you liked what you read so far and you have questions feel free to reach out on LinkenIn (stelios-moschos) | Twitter (smosgr) | hiretheauthor.com/steliosmoschossmos.gr.

Join FAUN: Website 💻|Podcast 🎙️|Twitter 🐦|Facebook 👥|Instagram 📷|Facebook Group 🗣️|Linkedin Group 💬| Slack 📱|Cloud Native News 📰|More.

If this post was helpful, please click the clap 👏 button below a few times to show your support for the author 👇


Kill your AWS Resources when you are done — the sequel was originally published in FAUN on Medium, where people are continuing the conversation by highlighting and responding to this story.