Getting Started with AWS Reserved Instances

If you've been using AWS for a while you've probably built up some excess spend. The "pay as you go" nature of AWS is a double edged sword: It's easy to get a PoC up and running, but you can wind up with waste if you aren't disciplined with your cleanup.

That's the situation we found ourselves in recently. My company has been running production workloads in AWS for 3+ years, and we've had 100% of customer facing workload in Amazon for over a year.

Over those 3 plus years we've redesigned out app delivery environment, released several new products, rebuild our BI environment, and reworked our CICD process. We subscribe to the "fail as fast as you can" methodology, so we've also started several PoCs that never went live.

All of that is to say we've done a lot in Amazon and we've tried a lot of new services. Despite our best efforts, we've had some wasted spend build up in our AWS bill. The whole engineering team was aware of it, but how do you start cleaning up waste, especially if your bill is large?

Sell the Marathon to Execs

Pitching a big, expensive cost saving project to execs is hard. Pitching a slow and steady approach is a lot easier. Rather than try to block a week for "cost savings" exercises we asked management for 1 hour working meeting a week. No follow ups for outside of the meeting, no third party reporting tools, and only low/no risk changes.

The risk with a dramatic cost savings project is that executives will think of it as a purchase rather than a continual effort. If they spend a lot of money on cost savings, they'll expect costs to automatically stay lower forever. If they get used to the idea of a small effort for a long time, it will be easier to keep up with it.

Start Small, Cautious, and Skeptical

Most of the struggle is finding the waste. Tools liked Trusted Advisor are useful, but they have, I hate to say, somewhat pithy recommendations. It's never quite as straightforward to turn off services as we might like. 

For example when Trusted Advisor finds an under-utilized instance you have a slew of questions to answer before you can turn it off. "Is it low use, but important?" "Is it used at specific times like month end?" "Is it using a resource Trusted advisor doesn't check?"

Instead of taking these straight recommendations pull a small coalition of knowledgeable resources into your 1 hour cost saving meeting. We started with

• A DBA - someone who knows about big, expensive systems who will be very cautious about damaging them
• An IT engineer - the team with permissions to create new servers who support customer environments (also very cautious)
• A DevOps engineer - someone with the ability to write scripts to cross data sets like EBS usage and CPU usage

With those three roles we had the people in the room who would get called when something breaks, meaning they would be very careful not to impact production.

Avoid Analysis as Long as Possible

Avoid getting bogged down with analysis until there are no more easy cost savings. With our cost savings team of a DBA, an IT Engineer, and a DevOps engineer we started with cost savings options that we all agreed on within 5 minutes. If we debated a plan for more than 5 minutes we tabled it for when we'd run out of easy options.

That approach let us show value for the 1 hour/week cost savings meetings quickly, and convince execs we should keep having them.

When you start to run out of easy ideas start doing more analysis to think through how much money you'll save with a change, and how much time it will take to do it. That will let you prioritize the harder stuff.

Document, Document, Document

Documenting your cost saving efforts well early on will lend itself to building our a recurring/automated process later on. If you save the scripts you use to find unused instances, you can re-run them later and eventually build them into Lambda functions or jobs that run.

It will also make it easy to demonstrate value to execs. If you have good documentation on estimated cost savings, actual cost savings it will be easier to show to your executives.

That's our high level approach, see my blog post on getting started with AWS Cost Explorer to start diving into your bill details!