Amazon AWS SysOps – Monitoring, Auditing and Performance part 3

8. Config Overview

Okay, let’s get into a very popular exam topic which is AWS config. So it helps with auditing and compliance of your AWS resources. So what does that mean? That means that we’re going to record configuration and changes over time. So as we start modifying the configuration of our ECU instances, EBS volumes, load, balancers, anything really. AWS config will record anything that was done over time and we record the configuration over time. So we can always roll back or see what it was a week ago. But the idea with Config is that we can help record compliance over time and that’s more importance. That means that we’re able to say whether or not the configuration of our resources is compliant with some rules that we’ll define in the future.

We’ll see how to do this in a second. Now you can get your alias config data and put it into S three. So as things are being changed and et cetera and recorded, you can analyze all that data using Athena. That’s just quite a nice integration. Now, what kind of questions can be solved using AWS config? Well, for example, is there unrestricted Ssh access to my security group? That could be one. Or do my buckets have any public access on this three? Or is my Alb configuration change over time and who changed it? How did it get changed? So the really cool thing is that with config we can track pretty much any resources in our accounts and view the configuration over time.

So the questions really that can be solved is unrestricted and is up to your imagination and needs. Now, anytime the config compliance rules, for example, is triggered, you can receive an alert, for example, an SMS notification, and you can configure any sort of integration you want as well with SNS. So you could react in real time to these compliance changes or configuration changes. Now, abuse config is going to be a per region service, but there is a way to be aggregating it across region and accounts so you’re free to use it. Multi account multiveregion it is quite nicely done as well. So let’s talk about config rules. Basically, AWS has a lot of rules.

There’s over 2075 at the time of recording, but they always add rules over time. Now, you can also make config rules that can be customed and defined in AWS Lambda. So for this, for example, we can say, oh, I want you to see all the EBS disk that are not of type GP two, they’re not compliant, or for example, each EC two instance that is of type two two micro is going to compliance, but otherwise it’s not going to be compliant. So you can just have whatever rule you can think of using a bit Slander, which is quite a nice integration. Now, the rules can be evaluated based on two different triggers. The first one is going to be every time there’s a config that changes.

So for example, every time my Alb configuration changes, I want all my rules to be evaluated. Or also we can do at regular intervals. We want to make sure that all our configuration is still the way it should be. Now, for pricing, this is where it gets tricky. There is no free tier. So this tutorial will be paid if you want to do along. Otherwise you just watch me and you’re going to pay $2 per active rule, per region per month. And after ten rules, there is a degree of pricing. So you can just look online. So this is something that matters because as we go ahead and create rules, then we’re going to pay and pass the free tier.

Now, what does it look like in the UI? We’ll see this in a second, but basically we can see the compliance of a resource over time. So here we can see our EC two security group, for example, was non compliant on the 24 December and then it became compliance seven minutes later. We’ll get to see this in a second. Or we can view the configuration of a resource over time. So seeing how an EC two security group has been changing the security over time, and then finally, if we enable the Cloud Trail API calls, if we track them using Cloud Trail, then we’ll be able to see an integration and be able to track down who did these changes on top of it. So let’s go ahead and have a look at how Config works.

9. Config Hands On

So let’s go ahead and open config. And here we’re going to be able to get started. And basically the service, as I said, is going to be able to provide an inventory of our resources and a history of all the configuration changes. And we can evaluate rules that will give us some compliance analysis. So let’s get started. The first thing we have to do is set the kind of resources you want to record. So remember, this is not a free tutorial, so if you just want to be in the free tier, swatch me too. But right now I’m going to say, okay, I want to record all the resources in this region. We’re going to also include global resources. If we wanted to record IAM resources.

For example, if you don’t want to take this box, you can say, okay, I just want to record auto scaling, scaling policy, or maybe EC Two, all these things under EC Two, et cetera, et cetera. For me, I’m just going to record all the resources just to show you the scope of all the possible things. And I will also include the global resources. Now basically, any time that we have a change, we can log that change into a bucket, and it could be an inrest bucket. So we’ll say Stefan arrest config, and we can also add a prefix if we wanted to do this, but for now it is fine. So basically any configuration history and configuration snapshot files will land into S three, so we can analyze it later.

Maybe using Athena, we could also stream any configuration change to an Amazon SNS topic, but we’re not going to tick this. And finally, because it is a new service on AWS and you need to do a lot of things, you need to grant it a service linked role. And so we’re just going to create it straight from this UI. Click on next. Now we get to choose the config rules. And so there’s a lot of config rules we can choose. And basically these rules will allow us to track the compliance ends of some of our elements. For now, what I’m going to do is that I’m going to skip this because I want to give you a good overview when we get there at the moment. So now we get into the review. We have no rules being created.

And all the things we do right now is just track all the resources configuration, I click on Confirm. And now Aos config is being set up. That may take a little bit of time. So I’m just going to pause until this is done. So now my resources are being discovered and that can take a little bit of time to be done. Okay, so my entire account has been mapped out. And as you can see, I have 71 resources in this region, which is really, really cool because I didn’t even know this right now. I know I have 71 resources and so I’m going to pay obviously, because I’m tracking these resources now. But all the configuration of all these resources is going to be tracked, which is also quite safe and quite nice to know.

Okay, so as we can see though, getting the resources information is not really, really great. If I click on one on Security Group and click on this one, well, I just get an information around what it’s like and I can get a configuration timeline. And here we go. That was the first time it was created, but there’s not much that’s happening really. I need to know over time if it’s compliance, if these security groups are open to the world maybe or not. So right now there is no compliance timeline because we don’t have created any rules. So what we have to do is go ahead and start creating rules. So let’s go in my dashboard and I’m going to click on Add Role. So here we’re going to add rules. And now we have a list of all the 82 address managed roles that I can add right now.

So you can see basically all these rules can apply to different services. For example, this one can apply to IAM. This one can apply to EC. Two, this one can apply to auto scaling. And so each rule has a description of what it does. For example, this one is really cool. It’s saying that if you like this rule, only the approved Amis will be compliant. Running instances that are not using this AMI will not be compliant. And so the really cool thing about it is that now we can just say, okay, there’s a bunch of Amis in our company, and we know we have the security software, we know they’re set up correctly, and if people don’t use these amis, we’ll know right away that there is a compliance violation and we can go and talk to people to migrate to the right AMI.

So that could be a really, really cool type of rule to set up. I’ll cancel it. The one I want to set up though is going to be the one on Ssh. So I’ll just type Ssh. And here it’s called restricted Ssh. So this is a very nice rule which shows whether or not the Ssh traffic can be completely unrestricted to any IPS, which is not what we want. So I’ll say, okay, I want to install this rule and I want to trigger this rule anytime a configuration changes. But we can also say periodic on some rules. For example saying every five minutes I want you to check, or every day I want you to check whether or not the rule is compliant or not. Now the scope of changes could be resources, tags are all changes.

So anytime a resource changes or attack changes, or both changes, then we can trigger a configuration change tag. And so for now, I’ll just say resources and I’ll say okay, anytime my security group changes, then you’re going to check whether or not Ssh traffic is allowed. So click on save. And now the rule is being evaluated. So if you remember I had something like seven security groups. So all the seven security groups are going to have their compliance evaluated against this rule and we’ll be able to get the results in a second. So I’ll have to wait for it to be done. And so here what we see is that out of my rules, three resources were non compliant and four resources were compliant.

So let’s just click on two of those to see the difference. So if I click on the compliant one, this one, well this was created using RDS and if I click on the configuration itself, so let’s go back to this compliant one. So this was the RDS Launch wizard and I can click on Manage Resources in the top right and get straight into the resource configuration. And so what I see is that yes, there is no 22 port here being open whatsoever, but if I go to my non compliant resource, so this one was non compliance with one rule, this was my Lunch Wizard Two. And if I go to Manage resource, what I can see is that probably in inbound we have Ssh on port 22 from anywhere and that makes my resource non compliance.

So what I can do is instead saying okay, maybe only my IP should be allowed, okay, not everywhere, but just my IP, I click on Save and now I should have made this group compliance. So what I can do now is just wait a little bit for this resource. And so automatically, because I’ve changed the configuration automatically, this should become compliant over time. So how do we see this? Well, we have two ways we can use the configuration timeline and the compliance timeline. So I’ve just opened both for the configuration timeline. As we can see here we have the first time my resource was being discovered, so this was on the 2 January at this time. And we can see here, we can see whether or not there were any changes and cloud trail events.

And so the information we get here is that we also get because we have enabled cloud trail in the past lecture, we can see that the root user did run instances, authorized security group ingress, create security group, et cetera, et cetera. So we can see all the Cloud watch events related to that security group. But now what we have to wait for now is basically for the configuration to change. Because if we go to the compliance timeline, as we can see right now, it’s not compliance, but because I just changed the security group rule after maybe five minutes, it just needs a bit of time to evaluate the rule again and to be triggered. I should be seeing the compliance so I’ll just pause the video for a few minutes.

So I just refreshed the page and it took a few seconds. But as we can see now, my resource is now compliant, so we get the exact time at which it became compliant. And the really cool thing is that if I go to configuration timeline I’m able to see that yes, some changes were made at 30 three and if I click on changes we can see the configuration changes that were done. So one rule basically disappeared. So this rule basically went away, which is the rule was added, so new permissions were added as a new rule and I only allowed Ssh from my IP. And so now we basically made this resource compliance and over time we were able to track the compliance and why it happened.

Thanks also as well to the configuration timeline. So the idea is that at scale, basically you’re going to have a dashboard in which you’re going to evaluate how many rules are non compliant and how many resources are non compliance and you’re going to set up as many rules as you need. It could be either the rules that are defined by AWS, which are quite handy, or custom rules that you want to define if you use AWS lambda. But by defining all these rules and on all your resources, you basically have a great account view on which rules are doing what. And how is your resource configuration changing over time for your entire resources on your entire AWS account, which I think in itself is really, really cool and nice, and I think something that should be enabled for everyone.

Unfortunately, you have to pay for it. So it’s just something you have to make a conscious decision for if you wanted to keep track of all your resources and all the rules you wanted. But definitely if you have a production a device account and you need some compliance rules or against whatever you want, then have a look at Config and set up as many rules as we can because overall it is really, really nice dashboard to know how many times you have noncompliant rules and how to make your resources compliant over time. So if you liked it, I hope you understand how Config works now and I will see you in the next lecture.

10. CloudWatch vs CloudTrail vs Config

So one very popular exam questions is to make the distinction between Cloud Watch, Cloud Trail and Config. Now, thankfully, thanks to the handson, hopefully you know exactly what the differences are. It’s pretty obvious in my opinion, but it’s never too bad to go through an example and see them. So Cloud Watch is for performance metrics, metrics CPU network and to create dashboards. You can also get events and alerts. And finally we have a log aggregation and an out tool if you wanted to. So Cloud Watch, I think we’re all pretty familiar with it Liz already.

Now, Cloud Trail could be new to you, but basically it’s to record API calls made within your account by everyone and everything, and you can define some trails for specific resources so you can get more information on EC two only and it’s a global service. Now, finally, Config is to record configuration changes and to evaluate resources configuration against compliance rules. Finally, you’re going to get a timeline of changes in compliance with this nice UI. So I think they’re a very distinctive service. I don’t think there’s any confusion, but let’s go through an elastic load balancer to see how each of these service can help you understand what is happening to your Elb.

So Cloud Watch can monitor the number of incoming connections, can visualize the number of error codes as a percentage over time, and maybe we can have a dashboard to get an idea of the load balancer performance. Maybe we can even make it a global dashboard if you have multiple load balancers for a global application. Now, config. Why would you use config on the Elb? Well, maybe you want to track the security group rules for the load balancer, making sure no one does anything fishy or changes anything. Maybe you want to also track the configuration changes for the load balancer itself to see if anyone modifies the SSL certificates or et cetera, et cetera.

We also maybe have a rule to say there always should be an SSL certificate assigned to the load balancer, and maybe we should never allow nonencrypted traffic into the load balancer. That could be two different compliance rules that you put into Config. Finally, Cloud Trail will be to track who made any changes to the load balancer with API calls. So in case someone changes the security group rules or someone changes the SSL certificate or removes it or whatever, then Cloud Trail will be how we know who made these changes. So all these tools are pretty complementary when you think about it. And when you understand that how they’re used for a load balancer, which I think is a great example, then you are going to rock any questions asked for you at the exam. So I hope that makes sense and I will see you in the next lecture.