Amazon AWS SysOps – Networking – VPC part 6

13. VPC Flow Logs + Athena

So now let’s talk about flow logs. Flow logs helps you capture information about the IP traffic that’s going within your interfaces. And you have three kinds of flow logs. You have the Vpc flow log and that applies to everything within your Vpc. You have the subnet flow logs which applies to something just within your subnet. And then you have the Elastic Network Interface flow log just for one network interface. So overall if you define a VPC flow log then it’s going going to have included the subnet flow log and the Euni flow logs as well. Okay, so why would you do this? Well, maybe to help monitor and troubleshoot connectivity issues in case some connections are rejected. We want to understand exactly why.

And so for this purpose, flow logs data can go directly into S Three or Cloud Watch logs. And when you enable it, it capture all the network information not only from what you own, but also for some of the AWS managed services interfaces such as Elb, RDS, Elastic Cache, Redshift, and Workspaces. So in our graph, what does it look like? We have a very complete graph, but now we’re going to add Vpc flow logs on the top right and the flow logs are directly being collected at the Vpc level or the subnet or the eni. But for now we’ll just say Vpc level and they go directly into Cloud Watch and or S Three. Okay, so now flow log, you are expected to understand how to read them. And so we’ll be looking at flow logs in this lecture.

But so it looks like this. There is a bunch of fields and there is version account ID, interface ID, source address, destination address, source port, destination port, protocol packets, bytes, start and action and log status. So that’s a lot of fields, but it’s important for you to understand the main ones. Source address and destination address will help you identify the problematic IPS or to filter by some IPS source port and destination port helps you identify the problematic ports and action will be success or failure. It’s also called accept or reject in the flow logs directly. And basically from this we can understand whether or not a security group or maybe a network ACL rule blocked our request. It can be used for doing analytics on your search patterns or observing malicious behavior.

And we should be seeing malicious behavior in this lecture. We’ll see that in a second. And there’s a tons of examples of flow logs at this URL. I recommend you do this in your own time to read it. And then how do we query Vpc flow logs? Because a CSV like some kind of like this format doesn’t really help us. Well we can use Athena on a three or Cloudwatch Log Insights and they’re really cool. We’ll see them both. So let’s get started. But to enable flow logs is super easy. You go to your Vpc demo vpc and then Flow Logs. And in there we can create a flow log. So let’s create one and we can set up a filter. So do we want to have all the accepted requests, all the reject requests only or maybe all the accept and reject? So for now we’ll just accept all and then the destination, it could either go to Cloud Watch Logs or to an S three bucket.

So let’s first do Cloud Watch Logs. So let’s open Cloud Watch. So I’ll go to services and I’ll type Cloud Watch and here we go. And I’m going to create a log group for it. So within it I’m going to go to Logs and I’ll create a log group. So let’s go back to Logs. Sorry. Action. Create log group and I’ll call it Vpc flow logs and I’ll create that log group. Excellent. So now it’s been created and I can go back to my Vpc and Flow Log creation page and I will refresh the log groups and I will say Vpc Flow Logs. The IAM role you need to select one to allow your Vpc to write to Cloud Watch. Thankfully you can just click on Setup Permissions here and this will automatically create an im role that has the required permissions. And you can view the policy documents right here.

Click on Allow and here we go. Done. So now we can go back to IAM role. Click on the refresh button and then here, scroll down and then we’ll find it at some point. Or I’ll just type in Flow in the search bar and we’ll find it for sure. Flow logs roll. Here we go. Okay. Click on Create. And now the following flow logs are created. So this one is created and it will go into Cloud Watch. Now this can take up to ten minutes to appear in Cloudwatch or in S Three. So we have to wait a little while, but for now let’s also create a second flow log. So we do Create Flow Log and this time we’ll send it to an S Three bucket. We’ll have all the filter and we have to enter a bucket ARN. So let’s go back to here, maybe this tab.

And I’m going to open S Three that has my service excellence and I’ll create a bucket. So let’s just keep something familiar. So I’ll call it Stefan Vpc Flow Logs Excellence and it’s going to be in Ireland and I’ll just go ahead and create it very quickly. So my bucket is not created, I’ll click on it and then I have a copy bucket ARN button here. So I’ll copy that bucket ARN and I’ll paste it right here. So you have to put the full ARN in there. Okay, and now it will say a resource based policy will be created for you and attach to the target bucket so that Vpc can send your logs to this bucket. OK, click on Create and the flow log has been created. Excellent. So now we have our two flow logs right here being created.

And if we quickly check into our S Three bucket, let’s have a quick look at it. We go to Permissions Bucket Policy, and this was added by AWS itself to allow our flow logs to write to this bucket. Okay, so now what we have to do is just wait a little while. What you could be doing as well is go to your instance, and you could be maybe going to the public one. And I’ll just curl google. com. Okay, this works. So you could just send a little bit of traffic just to make things moving. But don’t worry, even if you don’t do anything, traffic will come. So I’ll just wait ten minutes until we start seeing some data in S Three and in Cloud Watch also, where we wait, I will also restore Internet connectivity to my private route table.

So I will say, okay, on top of using the Vpc endpoint for S Three in case you go anywhere else. So here you’re going to use my Nat gateway.In this way, we also have traffic going into that second instance of mine. Okay, Excellence. So now let’s go to s three. Refresh nothing here. And then Cloud Watch. Oh, there are three enis already. So I’ll just scroll. Okay, so maybe we’re interested in the enis from my public instance. So let’s look at the right one. If we go to EC Two management, we find this public instance has the private IP ending in eight. And the network interface is this one. And it looks like the network interface ID ends with B 15. Okay, let’s have a look.

So we go back to Cloud Watch, and the restaurant won’t have anything on B 15, so I’ll have to wait. Here it is. This is the second one. And so we are getting a lot of information. As we can see, some of this traffic is reject and some of this traffic is accept. So it’s very, very interesting. It looks like some people are trying to access my EC Two instance because it’s public and a lot of traffic get rejected because it’s not authorized. So to prove this to you, let’s have a look at this one record. So two is the version of this flow log. This is my account ID. This is the eni that we have for this Vpc flow log, Excellence. And this is a source IP. So if we look at this source IP and we’ll do IP lookup, let’s have a look at what this IP is.

So we’ll go and we’ll type in an IP, get IP details, and it looks like this is a static IP coming from Japan. So someone in Japan is actually talking to my EC Two instance right now, but thankfully it’s rejected. And we can look at other IP addresses. For example, this one, let’s have a look at this one to see where this one is from, we’ll type it, look it up. And this one comes from Ireland. Okay, so there’s a lot of things happening right here. And what you should be realizing is that some people over the internet are scanning all the IP addresses and trying to find loopholes. So let’s go back to this first record. So someone in Tokyo is trying to target my institute instance. And then the source port that it tried to access is this one. The destination port was this one.

So it tried to access the port seven, six, six on my EC two instance. And that’s a bit scary, right? Then there is six. That means TCP. So you’ll have to look it up in the table. And then this is the start, this is the number of packets. Sorry. This is the bytes, this is the start, this is the end. And it was rejected and it was logged. So there is an okay, so super interesting to see that someone got an address request being rejected. But you could look at a lot of those. Basically, there’s a lot of IP addresses. Let’s look at this one, for example, on the Internet that will try to scan all your IPS on different ports to try to see if there is any flaw or something like this. So this one is in Germany and is trying to attack my EC two instance as well, but on ports 8088.

So everyone around the web, hackers mostly, are trying to scan for vulnerabilities and open ports. This is where you have to be very careful about the ports you open. So how do we analyze this at scale? Well, two ways. Number one is to go to s three. And so if I refresh this, I should be seeing AWS logs. And I go deep into it and within it I get access to all my Vpc flow logs as files. So I can download these files and keep these logs. Or we could use something like Athena. And this is a very popular question. So how do you analyze Vpc flow logs? Well, Athena will be the answer. So we have a default database, but we can create a new one. So let’s create a new table. So let’s go and type Athena DPC flow logs example.

And Google is going to be your best friend for this. So here we go. There’s a direct link and it says create a table. And then we need to modify the location. So let’s do this right now. We’ll go to Athena, we’ll paste this in. So create external table, if not exist Vpc flow logs. And that will go directly, I guess, in my default database. And then these are all the fields from my flow logs. And it’s delivered by a space and the location of it is and we need to specify the log bucket we have. So it’s Vpc flow log first defense. So let me copy this right here. Prefix. There’s no prefix. AWS logs. Then the subscribe account ID, which is right here. So I’ll copy this and paste it here. Vpc flow logs and then the region code EU west one.

Let’s verify this. Yes, it’s good. And click on run query. So now the query is successful and now I have a VPC flow log table. And the next thing you have to do is to go back to the documentation and add a partition. So let me copy this sent right here. And we go to Athena and we’re going to replace this entire thing. We’re going to add a partition, the location, I’ll just copy this entirely to gain time. And so the only thing we have to replace is the year, month and day. So this is quite manual. This is something you can automate using glue, but it’s out of scope right now. So let’s go to s three and 2019 110. So this is what we have. So 20190 110. And here 20190 110. That should work. Run the query. And now we’ve added a partition.

And so if we go back to the documentation now, we can for example, run this query and find all the reject on protocol equals six. So we’ll copy this, we’ll go back to Athena and then we’ll run this query. Run the query and as we can see in a second, after all the data has been analyzed, we get all these rows right here, which shows us the source IP address and all the reject and the protocol. So it could be really interesting to do some analytics. You can run any SQL query on your table. You could, for example, preview the table and this will show you all the rows in the table. Like right now there’s a limit of ten, but you could see all the rows in here and start doing some very interesting SQL queries. And that’s it. So this gives us a really good ways of looking at Vpc flow logs. I hope you enjoyed this lecture and I will see you in the next one.

14. VPC Flow Logs Troubleshooting for NACL and SG

So just a quick lecture on how you could troubleshoot security group and knackle issues using flow logs. So let’s look at an incoming request. So this is remember the flow of an incoming request. And so it turns out that if you get right away an inbound reject, it could be either a network ACL problem or a security group problem. Maybe this inbound rule blocked it or this inbound rule blocked it. And how do you know? Well, you look at the EC two security group inbound rules and figure out whether or not this is a match for the request that was done, else it is a network SEL problem. And then if you get an inbound accept so if these two queries go through and then you get an outbound reject, well it turns out that it can only be a network SEL problem because security group are stateful and so because the inbound rule is allowed, then the outbound rule is allowed as well.

So for outgoing request, same idea. Basically if you get an outbound reject so if this outbound right here doesn’t work, it could be either a security group issue or a network SEL issue. And then if you get an outbound accept so this goes through but then you get an inbound reject, it could only be a network SEL problem because your security group is stateful and would have allowed the request back in. So this could be a very common exam questions as well. They will show you two Vpc full logs line and you have to understand how to troubleshoot those. So remember this diagram, remember what how they work and remember about Vpc four lugs and you’ll be all fine. All right, I will see you in the next lecture.