Amazon AWS SysOps – EC2 Storage and Data Management – EBS and EFS Part 4

14. CloudWatch & EBS

So for Cloud Watch and ABS, you need to know a few things. You need to know a few metrics. And it’s going to be volume idle time, and that’s the number of seconds. Then there is no read or write being submitted. That means that your disk is idle and doing nothing. Volume Q length, which is number of operations waiting to be executed. And so if you get a really high number, that means that a lot of operations are being queued. That means that maybe you don’t have enough IOPS or maybe there’s an application issue. So it is definitely something to know about. And the burst balance as well that we talked from before. If it becomes zero, then you may have problems sustaining your application throughput and you may need a volume with more I Ops overall.

So another thing to understand is that your volumes metrics are reported every five minutes for GP Two and every 1 minute for IO one volumes. So just something to know. If they ask you about the volume interval, it’s not something you can enable on the console just the way we could for EC two have detailed monitoring. For EBS, you just get one option based on the volume type and then EBS volumes where they have status checks just the same way EC two instances have status checks. So, okay means that the volume is performing good. Warning is that the performance you get out of your volume is below what you expect impaired. That means that you have severe performance degradation.

That means you need to act on it. And insufficient data means that the metric collection is in progress. So let’s go have a quick look in the AWS console. So in the console, if we go to our volumes and we go take our one of our volume, for example, this one, the eight gigabyte one, and we go to monitoring here we see all our metrics.

So we can see the read bandwidth, the right bandwidth read throughput, right throughputs Q length and time idle, percent read size, right size read lense, right lengthy, and burst balance. So I think all these metrics are super explicit around what they are. The one thing I said was important was around time idle spends. So how many percentage of the time your disk is idle and 100% means it’s unused. Zero means that it’s always busy. Average queue length, which says how many operations are queued on your disk. So right now we’re very close to zero.

So it’s perfect. But if you get a very, very high number, that’s a problem with your disk. Here you get an information around how you’re using your Ups. So, for example, here I say I use 5 Ups. And remember, we can go all the way up to 256 IOPS. Yes, and we got to end the burst balance. We already saw this. 100 means that we’re stable and good and we have enough burst to go through unexpected IOPS load. So that’s it. Just remember these for the Cloud Watch metrics, obviously.

And now if you go to the status check here, you get some information around the volume status, which is okay, and you’re AZ and then you get more information. But that’s out of scope for now. The only thing to remember is that you get a volume check that is happening, and so you could get more information about volume status check in events. Okay, so that’s all for your monitoring. I think for Cloud Watch it’s pretty easy. But just remember the main metrics that are going to be for troubleshooting. So average queue length, time spent, idle and boost balance. And I will see you in the next lecture.

15. EFS Overview

Okay, so now let’s talk about EFS. EFS is a service you need to know at a high level going into the exam. But it is very interesting from an architectural standpoint and has a really amazing set of features. So what is EFS? It stands for Elastic File System and it is a managed NFS or Network file system that can be mounted on many east instances across many different Availability Zones. So as I said, it works with mulitas and this is a huge difference of EFS versus EBS. EBS was locked into a single Availability Zone, whereas EFS is going to be mountable across multiple Availability Zones. And so as such it’s highly available, it’s scalable, but also extremely expensive.

It’s about three times the cost of a GP two drive. But you only pay for what you use. So if you don’t store that much data, it makes you be cheaper to use EFS than EBS based on how well you manage your data set and size on your EFS drive. So here’s your EFS and this is a network file system and you attach a security group to it to manage incoming connections. And so you have different EC two instances across multiple AZ. So USD One A, USD One B and USD One C and they will all be mounting the same NFS, the same EFS onto their file system and they will all access the same files. Okay? So EBS was something that was linked to one easy two instance at a time.

And so the data was not shared between multiple EC two instances. But in this case with the EFS it’s a network file system. And as such, all the easy two instances have access to the same files onto your EFS drive. So use cases for this well, content management, web serving, data sharing or a WordPress website. Now you need to know it uses the standard Nfsv 4. 1 protocol. So it is a standard way to mount a network drive. And to access the EFS file system you need to use security groups. So this is a network security. EFS is only going to work for Linux based AMI, so not Windows. So this is something that’s extremely important. Windows instances cannot mount an EFS onto their file system. To encrypt the EFS, you can use Kms keys at rest. And so again, as I said, EFS is going to be used only for POSIX file systems. So basically Linux and has a standard file API and the file system will scale automatically. It’s paper use, no capacity planning.

So that makes it a very easy offering to use. Now, let’s talk about some configurations that you have for EFS and they’re important to understand going into the exam. The first one is around the scale. So EFS is built for thousands of concurrent NFS clients. So thousands of EC two instances mounting the same NFS drive at the same time and also has a massive scale. We’re talking about 10GB plus per second of throughputs the way AWS advertises EFS is that it can grow to petabyte scale network file system automatically. So that makes it something truly big in terms of performance mode. You have two ways of setting it and you set it at creation time.

We have general purpose, which is when you have the default latency set of use case. So we have a web server, a CMS, et cetera. So this is going to be the default one for EFS, but also max IO. And this is going to give you more throughput. It’s going to be more highly parallel, but it’s going to be having a higher latency. So this is well suited for if you have a big data workload media processing. So big files, huge files, not many small files and they can be accessed with a bit more latency. So do you need to be able to choose between real purpose, usually for anything web related from max IO, which is going to be anything processing related and then extremely important. And you have to remember this, there are different storage tiers for EFS.

And so you have a lifecycle management feature that allows you to move files between a different tier to another tier after n number of days. So the standard storage tier is for frequently accessed files. So files are going to be frequently requested onto the file system and they’re going to remain into this standard storage tier. But in case you have some files that are infrequently accessed, you have the infrequent access storage tier on EFS called efsia and that can come up at the exam.

And the idea is that the files are going to be less accessed, so it’s going to be having a lower price to store these files. But in case you do need to retrieve these files for whatever reason, then you’re going to have a retrial fee and there will be a small cost. Okay? And so again, you need to remember maybe some files are going to be better off to be on the standard storage tier and some files will be better off in the infrequent access storage tier. And that’s it for for EFS. I hope you like this and I will see you in the next lecture.

16. EFS Hands On

So let’s go and create our first EFS network file system. So let’s type EFS and we are in the EFS console. So let’s create a file system and as we can see we have a very simple dialogue and you can click on Create and it will just go ahead and create it but we want to go through the options so we’ll click on Customize to look at all the options for our elastic file system. So the name is optional so we’ll leave it empty. We can enable of automated backups to just have a backup of our network for our system which is nice. And we have a lifecycle management tab. So here we can use something called the EFS infrequent access storage class and the idea is to say okay, if a file has not been accessed in 30 days, it looks like it’s a file that is infrequently accessed, therefore move it to the efsia storage class in order to save some cost. And that makes sense.

So you can say seven days, 14 days to 30, 60 or 90. Okay, so we’ll just leave it as a default of 30 days. Then we get different performance modes. So we have general purpose or max IO. And General Purpose as the name indicates is ideal for latency sensitive use cases such as web serving environments and content management systems. So if you have a WordPress for example, this would be a great use case and Max IO scales to a higher level of aggregate throughputs and operations per second with a bit more latency. This is better for a big data or processing file processing type of use case. So we’ll just leave at general purpose for now. The throughput mode can be either bursting and that means that the throughput will scale with the file system size, allowing you for short burst or if you want deterministic throughputs for your EFS file system.

 Maybe because you know you have a small EFS file system, but you need high throughputs. Then you can provision how many megabytes you want up to 1024 MIB. So megabytes per second. We’ll just leave it as bursting as well. We can enable encryption at rest for our EFSL system and scroll down. Now let’s click on next. Very important now are the network access settings. So we are operating in our VPC and we can mount it across multiple different availability zones. So EFS is a network file system and we can have it across different AZ as I’ll demonstrate you in a second. And so for each AZ you should define a security group. And so right now I’m going to go ahead and create the security group we need. So let’s go into the EC two console and I will go into the Security groups tabs on the left hand side create a security group and I’ll call this one my EFS demo and for now no inbound rules and I will go ahead and create the security group.

Okay, so security group for EFS and created. So now we’ll use my EFS demo into this dialogue. So I will remove all these security groups and I will choose my EFS demo which I probably have to refresh this page. So let me do this right now. I’ll refresh this page and very quickly scroll down, click on Next. And here we go. So I will choose now my EFS demo right here to be the security group for each different attachment point. So, okay, and the last one my EFS demo good. So we’ll see what that security group impacts in a second. Then I will click on next. File system policy is optional and this is out of scope. So I will just go ahead and skip this. And finally we can review everything. So we can review and create. So everything looks good here.

We have encryption, we are in our VPC, we have IA enabled. We have three availability zones that are going to work with our EFS file system and they all have the same security group that we just created. And I will just go ahead and click on Create. So now my file system is creating and while that happens, I can go ahead into my EC two and I want to create two EC two instances that are going to be accessing that EFS file system. So as we can see, let’s go ahead, the file system is created. It is created and so if we look at the size we can see that we are using 6. We’ll only pay an EFS for what we’re using. So we’re using 6 KB. So this is what we’re going to pay for and we can go also to get some information around the size and EFS infrequent access. So how many files have been moved into that lower, much lower priced tier in EFS? Okay, so everything looks good right here.

Now let’s go ahead and create our EC two instances. So I’ll click on Launch instance. I will choose Amazon Linux two AMI T two micro to remain within the free tier. And then I will choose one instance and I will launch this one in EU West two B for example, as my first AZ. If you scroll down you can see that there’s file system and you could add your EFS file system here, but we’ll not do that. I want to show you how it’s melted. So don’t click here, we’ll click on Add Storage. We can leave the storage as is. Add tags. This is fine. I’ll go ahead and create a new security group for my instance and I’ll call it EC Two EFS because this is my EC two instance that’s going to access my EFS network file system. We’re going to allow SSH review and launch.

And yes, I have this key pair launch my instance. So now this instance is launching and I’m going to launch a similar one but in a different Availability Zone. So I right click instance sorry, launch more like this. And then in my instance details. I will edit the instance details, and I will set it in EU West two A to be a different Availability Zone. Review and launch, launch and launch this instance. So here we go. Now, we have two instances that have been launched in two different Availability Zones of this one and that one. And we want them to be able to access our EFS network file system. So I’m going to SSH into each of these instances. So this first one is right here.

I’m going to launch my SSH command so easy tutorial PM and an Easy to user at my IP. I’m into my first host, and I’m going to take this IP of my second easy to instance and launch a similar command. So here we go. Easy to user at the IP. Okay, so I’ve done SSH into both instances, and they’re both in two different Availability Zones. Next, I need to install EFS on these instances. So the easiest way is to go back to the EFS console, and on the top right there is Attach. And this gives you some information around how you can attach EFS into your instances. So, as you can see, we can mount via DNS or mount via IP will use mount via DNS and we’ll use the EFS mount helper.

So to use this thing, we have to go into the User guide in the documentation and install a small package onto our EC, two instances called the Amazon EFS utile package. So we click on installing the Amazon EFS utile package on Amazon Linux Two. And as we can see, we can scroll down and do this pseudo Yam install command. So let’s go ahead and do this Yam install command onto both my instances. And this is going to install the necessary packages to use this EFS mount helper. Okay, so this was very quick. Now back into EFS. We need to create the EFS directory. So to do so, very simple. We’re going to do makedir EFS and makedir EFS. So now if we look into both of our instances, they both have an EFS folder. Next, I’m going to run this command right here to mount the EFS drive using TLS. So they will be in Flights Encryption, and I will mount it into this EFS directory.

So let me copy this command right here and paste it. Press Enter. And as you can see, there is a timeout that’s because we need to modify the security group settings. So let’s stop this command. And I’m going to go into my EC to console. And we need to modify one security group. So if we remember, we have attached a security group into our EFS network file system, which was this one. Myefs demo and currently my EFS demo in terms of inbound rules, does not allow anything. What needs to happen is that my EFS demo needs to allow inbound from EC two to EFS. So very simply, let’s edit the inbound rule, add a rule and we’ll look for NFS, and the source of it is going to be EC two to EFS security group. And so we allow EC two instances into EFS, we’ll save this rule. And now that this rule has been done, we should be able to go back into EFS, try this command again, and then it should succeed.

And it has succeeded. I can apply the very same command in here on the right hand side, and it has succeeded as well. Okay, good. So what did happen? Well, let’s go into the EFS directory. So I just changed directory into the EFS directory, and currently we can see there are no files. But what if I create a hello world. TXT file in here? Well, I don’t have enough permission, so I will do pseudo touch, hello world. TXT. That should do it. Now, if I look into the files, I have a hello world. TXT file here. And if I look now on the right hand side and list the files into EFS, we can see the same hello, world file has been created. So if I do pseudo echo, if I do pseudo nano hello world. TXT to just edit and say hello world from the first instance and save this file. So if we look into the content of this file, hello world. TXT, it says hello world from the first instance. And if we look into the content of the very same file onto the right hand side instance, also hello world from the first instance.

So this file system mounted on the left hand side and the right hand side is the same. It is a shared network file system. And that is the whole power of EFS. And that’s it very simple. So when you’re done with this, you can go ahead and delete the EFS file system, obviously, and you could go ahead and terminate these easy two instances if you needed to. So you click on these two instances, action and then terminate, and you’ll be good to go. So that’s it for me. I hope you liked it and I will see you in the next lecture.

17. Section Cleanup

Okay, so let’s just clean up this entire section. So for this, I’m going to go to action on my file system and delete it. And for this, I have to enter the file system ID. So I’ll just copy and paste this. And so okay, we’re going to get rid of our file system. Perfect. Now into our EC two instances. Make sure to terminate any running easy to instance. So that’s good.

Then in terms of volumes, we’re going to have to clear up volumes as well. So any volumes that is available, you have to terminate it. So I’ll right click and delete all these volumes. Okay, now for snapshots, you’ve made a few snapshots, so let’s just go back and delete every snapshot you have just so that you can get rid and not pay for any storage for snapshots. And then finally security groups, if you want to, because it’s going to get a lot of security groups right here.

You can basically delete so many of these security groups except the one for default. Just don’t delete that one. So you can go ahead and delete these security groups, and they will only be deleted once. You have basically deleted all your EC two instances associated with those. So maybe these won’t be deleted right away, but you can just keep on trying until it works. I’ll delete the security group now. Yes, this one works. I’ll delete the load balancer security group. Yes, they worked.

And this easy two for EFS is still used by my instances. So I have to wait for my instances to be shutting down properly before I can go ahead and delete that last security group. So I’ll have to wait a little bit, but that’s it. Once you’ve cleaned up everything, you’re ready to go for the next section.