VPC Flowlogs through Cloudwatch Logs Insights

You know all those times you think to yourself, "I wish there were a faster way to search all these logs I keep putting in Cloudwatch?"

Well apparently Alexa was reading your mind at one of those times because at AWS re:Invent 2018 released CloudWatch Logs Insights. It's advertised as a more interactive, helpful log analytics solution than the log search option we already have.

This last week I got some questions about blocked traffic in our AWS account, which seemed like the perfect opportunity to give it a shot (NOTE: You will need to be sending your VPC flowlogs to Cloudwatch for this to work for you).

Here are some of the queries I tried out, most of them based loosely off of the examples page.

Count of Blocked Traffic by Source IP and Destination Port

fields @timestamp, dstPort, srcAddr
| stats count(*) as countSrcAddr by srcAddr, dstPort
| sort countSrcAddr desc

This query gives use the the top blocked senders by destination service.

Using this I pretty quickly found an ELB with the wrong instance port.

This worked for me because we separate our accept and reject flowlogs. If you keep them together you can add a filter as the first line

filter action = 'REJECT'
| fields @timestamp, dstPort, srcAddr
| stats count(*) as countSrcAddr by srcAddr, dstPort
| sort countSrcAddr desc

Blocked Destination Addresses and Ports for a Specific Source

During our troubleshooting we noticed a specific address sending us a lot of traffic on port 8088, which is vendor specific or an alternate HTTP port. This was a little odd because we don't use 8088 externally or internally.

We dug in using this query

filter srcAddr = 'x.x.x.x'
| fields @timestamp, dstPort, srcAddr, dstAddr

Where x.x.x.x is the source IP address.

I'm not going to post the screen shot because there are a lot of our destination addresses and it would take time to block them out, but you get the idea.

We did a lookup on the address and it was owned by Digital Ocean, which is a cloud hosting company. It's likely someone was doing a scan from a server in their environment, hard to say if it was good or bad intentions.


To satisfy my curiosity I wanted to ask the question, "When did the scan begin and how aggressive is it?"
So I added a "stats" filter to group the sum of the packets by 5 minute totals.

filter srcAddr = 'x.x.x.x' and dstPort = '8088'
| fields @timestamp, dstPort, srcAddr, dstAddr
| stats sum(packets) by bin( 30 min)

When you use the stats command with a time series you can get a "visualization" like the one below:

It looks like the scan lasted about 6 hours, from 4 am or so my time to around 9:45 my time.

Conclusion

CloudWatch Log Insights is a much faster way to analyze your logs than the current Cloudwatch search. The query language is pretty flexible, and reasonably intuitive (though I did spend several minutes scratching my head over the syntax before I found a helpful example).

While it's an improvement over what was there, it's not on par with a log analytics tool like Splunk, or a data visualization tool like Kibana. The visualizations page for Insights only works with time series data (as far as I can tell) and isn't very intuitive for combining multiple query results. For that you still have to import the query into a cloudwatch dashboard.

Amazon's edge over those more mature tools is that it's integrated into your AWS account already, with almost no setup required (except getting your logs into Cloudwatch), and the pricing model. As usual with AWS it's extremely friendly for getting up and started, but it's easy to see how the cost could grow out of control if you're not paying attention (picture someone creating a query for a months worth of data on a dashboard that refreshes every 30 seconds).

Happy querying!