Amazon AWS SysOps – CloudFormation for SysOps Part 2

11. CloudFormation Intrinsic Functions

So the exam may ask you about interesting functions and we’ve seen them, but here is a list of the ones you should absolutely know for the exam. The ref function, the get attribute find and map, import value, join sub and the condition functions. So let’s go ahead and have a quick memory refresher on what they are. The ref function is to be the most important. It used to be referencing parameters and in case you reference a parameter, it will return the value of that parameter. So we’ve seen this in practice when you had the security group description and resources.

So if you reference another resource in your cloud formation template, it will return the physical ID of the underlying resource. So for example, if we reference an EC two instance, we will get the EC two instance ID. Now, the shorthand for this is going to be exclamation point ref and this is what it looks like. So here is a quick example. We create a subnet and for the properties VPC ID, we reference the VPC that was created from before and so what we get out of it is the VPC ID. As I said, when you reference a resource, it returns to physical ID.

So super important to know. Now, how do we get other information out of resources, right? Because if we only can get the ID information out of the resources using a ref, then we’re pretty much stuck for some things. So we can use get at and attributes can be attached to any resources you create. And so to basically get a list of all the resources and all the attributes that get exposed from these resources, you will have to look in the documentation. So let’s do a quick look at it just to get an idea. So if we look at our EC two instance and we go to return values, we can see that there’s a ref. So the ref will return the ID of that instance. Okay? And then using the get attribute function, we’re able to get the availability Zone, the DNS name, the private IP and so on.

So all these things are exposed. So to know what attributes are exposed in each resource, you have to go to the resource and look at the documentation. Now for example, if we want to get the AZ of an Et two machine, we saw we could do it. Here is our resource block. So we have our EC two instance with an image ID and an instance type and we want to get the AZ out of it. For example, when we create an EBS volume for this, we’re going to create a new volume under resources and we say the type is an EC two volume condition if we had a condition from before.

But the important thing here and there is that if you look at the properties, the size is 100 and the AZ is using the get attribute function EC two dot availability Zone. So the EC two instance is coming straight out of this name on the left hand side and then the dot says it tells get attribute what we want to get out of it. So we want to get the Availability Zone. So this is quite a popular exam question as well. How do we get this attribute of this resource? And the answer is use the Get at function. You have Find in Map and we’ve seen this before.

We use this finding Map function and the shorthand syntax is this we have to specify the map name, the top level key and the second level key. And so here is a quick summary. In this example we were using the Find in Map function to find the image ID based on the region we were in and the type of architecture 32 or 64 bits. Import value is to import values that have been exported as outputs in other templates and for this we’ll use the import value function. So again, remember when we import a value we just give the import value name that was exported and it should work just easy. Now for join join, we haven’t seen it yet but we can join value with the delimiter.

So this is the shorthand syntax we should define join. We give the delimiter name and then we provide a comma delimited list of values. To make it slightly more concrete, to create the A column B column C string, we can use the join function with the column and then we specify ABC on the right hand side. So if you do programming, this is quite a common function in programming, so just know about it sometimes maybe you’ll be asked hey, what is the output of this interesting function? And if you use join well, you know that you have to just put the columns in between the A and the B and the B and the C. Okay? Finally we have the sub function.

The sub function is a shorthand for substitutes. It’s super handy, it allows you to substitute values within strings and so you can use it with references or pseudo variables. And the string must contain this dollar sign and open with the variable name to do a substitution. So this is a bit obscure, but just remember sub does substitution and it will totally make sense when you see it, if you see it in the exam.

Okay? Sub is to substitute values. Finally, conditions is when we define for example, a condition to only create the prod resources and we can use a bunch of functions in there. And so the interesting functions we can use is and equals if not and or and maybe others that will get added to confirmation over time. But that’s all for the intricate functions you should use. You recognize the function is called intron when there is this FN column or when you have that little exclamation point before a name that represents a function. This is all you need to know for Cloud Formation functions. I hope that was helpful and I will see you in the next lecture.

12. CloudFormation User Data

All right, let’s get advanced with cloud formation. Now let’s talk about user data. So we know we can have user data when an easy two instances launches and we can set it through the console. But there is also a way for us to include this in cloud formation directly. And so for this we’ll pass the entire script through the function called FN base 64 and that will convert our script to a base 64 string. That is what user data needs in Cloud Formation. So the good thing to know is that when you run the script, all the log and that’s just not related to confirmation, but just user data in general. All the log of the user data will be in Varlogcloudinit login. We’ll see this in a second.

So let’s see how we can do this in confirmation right away. Okay, so I’m in confirmation. Make sure you’re in North Virginia region as well. And there’s a new redesigned abuse cloud formation console. It doesn’t change much, but we’ll just click on it just to make sure that it looks like you may get to see in the future. All right, so we have this new UI. It’s exactly the same thing, don’t worry. So let’s go ahead and we’re going to create a stack so the template is ready and we’ll use and upload a template file. But before that, let’s go and see what the template is. So we’ll upload three user data. YAML. And so if you look at this template, it has one parameter, which is the SSH key we’re going to use. So make sure you have a key pair available in US East One.

If not created, then we’re going to create an instance and it will be in US East One A. It will use this AMI. And this is why we’re blocked to use US East One A is because AMI is a region based and then we’ll have a key name refss key. So we’ll be able to SSH into our instance. And then we have an SSH security group which allows us to SSH on port 22 and also view on port 80 because we’ll install a web server. And so the important thing here is user data. As you can see, we pass through the function, interesting function, base 64. We pass a script and this whole script is preceded with a vertical pipe. So the vertical pipe here means that this whole thing is one giant string with new lines. If you delete this, it won’t work. So you need to have that vertical pipe to say all this is a script. And so in the script, very simple. We’ve seen this from before. We install Httpd, we start the system and then we say hello world into index HTML. So that’s about it. Let’s go and have a look at how we can do it.

So we upload a template file. So I’ll choose it and it’s user data and then I click on next and so the stack name here, I’ll just call it user data example and SSH key. If you do have a SSH key it will show up here. Otherwise you can go in EC two. So you go to service EC two and then on the left hand side of EC two you would go all the way to key pairs and in key pairs you could create a key, name it whatever you want and then it will download it for you. So for me I’ve already done that. So I’ll just use this confirmation key pair that I have. I’ll click on next and then I’ll just won’t specify any option. Click on next. As you can see the template is an s three and the parameter is SSH key equals confirmation key pair. Everything looks good and I’ll click on create stack.

So let’s go ahead. And now we’re just going to have to wait for our EC Two instance to come up and ensure that the EC Two user data get applied. Okay, so it looks like my stack is now completed. So if I go to resources and refresh I can see that there is my instance in a security group that was created. So I click on my instance and that will take us straight to the EC Two management console. And in there we can look at the public IP. For example, here it is. And if I go to the public IP URL, I get a hello world from user data. The cool thing I want to show you is that we can SSH directly into this instance and we can see that the user data get executed by doing cat VAR log and then cloud init log. And this is where all the user data output will go. If you press Enter you can see that for example, this was the command when we were running a yum install httpd.

So everything got installed and we get the output right here. And then with systemctl start httpd enable httpd and then Hickok ll you from user data and then it says cloud in it has finished at this time. So if there was any issue in our EC to user data script, we would look into that file again. We would look into verge cloud init output log and in that file we will discover where the error was. Which is quite nice when you debug. But something to think about right here is that we created our stack, but our stack was create complete working even though even if our user data did not work, we would still have gotten a good green tick. So we have to think about how do we solve that problem. And we’ll see this in the next lecture. But for now you’ve learned how to use user data with confirmation. Congratulations. I will see you in the next lecture. And don’t forget to always delete your stacks right after you’re done.

13. CloudFormation cfn-init

All right, we’ve seen user data but it’s pretty clunky as soon as you want to have something quite complex and quite structured. So for this we’ll use CFN in it and there is this block in the metadata of the resource that we’ll see in a second called Cloud Formation in it. And if you include it you can use a script called the CFN in It script which stands for Cloud Formation in it and it helps make basically complex easy to configurations readable and the easy to instance will query our Cloud Formation service to get the init data and then apply it. And then as soon as it applies it, all the logs will go to the VAR logcfninit log file and we’ll see it in a second.

So why does it matter? Well, you’ll see in a second I’ll show you. It looks quite more readable than our script. So we have the confirmation service and it will launch an EC two instance. On this EC two instance as part of the user data script I will run CFN in it and the action of such will make the easy to instance query confirmation service to retrieve the init data and then it will apply it. So let’s see how this works now through an example. Okay, so as we can see in three user data YAML, we had this giant script base 64 and it was a simple one but it could get way more complicated. Think of like all the things you want your EC two instances to do when they start. So now if you go to CFN init YAML, we have the same parameters and SSH key and we have an instance as well running the same AMI with the same key name and the same security groups. But now the user data is going to be a little bit different. It’s still a base 64 but now we on top of it passing through the sub function. We still have that vertical pipe and here we’re going to do first thing update the CFN bootstrap script to get the latest version and this is just something you have to do and we have them installed already because we use a Linux AMI, Linux two AMI and then we’re going to start the CFN init script. So we’re going to run optimist bin Canine.

Then we pass in the stack ID. So this is a stack ID that represents a pseudo variable of our stack. Then what we want to retrieve so my instance, we want to retrieve the CFN in it metadata from my instance which is right here on line 26. So we’re going to retrieve this metadata, we specify the region which is a region of the stack and then in case of error we’ll say exit fail to run CFN in it. So the metadata itself, oops the metadata itself is going to install a simple Apache Http page and it has the AWS confirmation in it block and in there there’s a config.

And now we can read that config through. I won’t walk you through how I wrote this. The important thing is to know how it understands for the SIS Ops exam. So here we have packages and we say okay, from Yum packages I want to install Httpd. So here we’ve just said install Httpd and we should do the exact same thing as when we did Yum install Httpd before. So we’re good here. Then we have files and File is saying this file at VAR www HTML index HTML must have some content, the content being all these things. So here, as you can see, I had more stuff at h one block, a p block, and it was easier because I could use that vertical pipe again, whereas in the user data script I had to use this echo thing, which was a bit clunky. So here we have more content, more freedom. We can also set the mode of the file to be equal to 64 four. Then we can set some comments, we can say hello and we’ll just run the comment echo hello world.

And then finally we can say services should start and we say okay, the Httpd service should be enabled and we should ensure that it is running. And so the cool thing about this cloud formation in it is that I think it is way more readable than the user data script, which is a basic bash script. And so this is why we would use Cloud Formation in it. The rest is the exact same thing. But let’s have a look at how this works in the console. So here’s the console and we’ll go create Stack. Template is ready, we’ll upload it and we’ll choose the file Cfninit YAML, I’ll click on Next and I’ll call Cfninit example and the SSH key, I’ll use the same key as before next and then create Stack. Okay, now our stack is created and I’ll just wait a minute. Okay, so my stack has been created and if I go to my EC two management console and refresh and see all my instances, this one has been running.

So I can go to the public IP and make sure that the CFN in it worked and it says hey, hello. Well from EC two instance, this was created using CFN in it. So pretty cool it’s working. But the cool thing I want to do is to SSH into the instance and see what happened. Exactly. So let’s just go and SSH into this instance and we’ll have to remove it, receive the Http copy and paste mistake. Here we go. We’re inside of it. So now, as I said, there is the far log cloud init that we can look at. And as you can see, we just run this command CFN in it in there and we don’t get really any output out of it. We don’t see anything. So to see the output of this command, what you have to do is look at the file named VAR log, Cfninit log, and this file will contain the output of our CFN init script.

So, as we can see here, it’s installed Httpd, it did the command hello, it enabled the service Httpd, and it started the success of the service successfully. So in this basically VAR log directory, you can see a lot of things. You can see all the logs that come out of the CloudFormation things. And so it’s pretty cool. You can also look at this CFN init command log that felt that could be interesting as well. But the idea is that you need to understand that if you needed to troubleshoot everything, they would be right here. And here we get the actual outputs of all the commands which if wanted to debug for whatever reason, we could. Look at this.

So yummy, install Httpd, it worked very fine and then at the very bottom command hello world it said hey, hello world and it completed successfully so that’s what I found really cool. Now we have more structure and we have more files to look at in case of things go wrong and I think this is a better way to configure your instances. So the only problem we have is that this instance still is being created. So we have still create complete. But it’s doesn’t matter if our CFN init script works or not. What we’d like to do is for the create complete to be done only when I know that this has worked. So how do we do this? Well I will see you in the next lecture.

14. CloudFormation cfn-signal and wait conditions

Okay, so as I said, we still don’t know how to tell confirmation that the EC Two instance got properly configured, that it was successful after doing a CFN invite Maybe it failed, maybe we don’t want confirmation to complete because our configuration of our instance has failed. And so for this we can use the CFN signal script. And so for this we’ll run the CFN signal command right after the CFN init command and we’ll tell confirmation service to keep on going or fail based on that outcome. For this we’ll need to define a wait condition and the wait condition will tell the cloud formation templates to wait until it receives a signal from CFN signal. So we’ll block the template until it receives a signal and we will attach a creation policy basically that works also on EC Two and ASG. But the creation policy will dictate how many signals we want to receive and for how long we’re willing to wait. No worries, we’ll see the code in a second so keep on hanging.

But in terms of the diagram, you need to visualize how it works. We have the cloud formation service and it will launch an easy two instance just like before. This time though, there will be a wait condition. So the wait condition will just wait and put the confirmation service in hang mode and it will wait until it receives a signal. So in the meantime, the EC Two instance will run the CFN init script and so we will query the cloud formation service to retrieve the init data just like it did before. But then after it’s done we will run the CFN signal script and we will signal the wait condition that we’re done and then confirmation will be happy and we’ll be able to move ahead. So let’s go how this works in practice. So in Cfninit YAML, as we can see we ran the CFN init file but that was nothing afterwards.

So let’s go to CFN signal and we’ll look at this. So the exact same parameters, the exact same MYAC to instance. But now if you look at the user data after the CFN init command, we’re going to run the CFN signal command and it’s basically going through this little minus e dollar question mark. So this is special bash but it basically says if this works then send a good signal, if this didn’t work then send a bad signal and we’re going to signal our stack and we’re going to signal our weight condition. So the simple weight condition that was defined right here in the bottom, I’ll go over it in a second in this region. And so this is saying if this worked, signal the weight condition that it’s all good, if it didn’t work then still signal the weight condition that it didn’t work and then cloud formation will draw its conclusion.

The metadata itself hasn’t changed. It is the exact same cloud formation in it. So because it worked before, it should work now too. Okay, let’s scroll down. And here is our weight condition. So our weight condition of the type of it is a cloud formation weight condition. And the creation policy says that we’re waiting for a resource signal and the timeout is going to be and I’ll just set it to two minutes, just to be fair. So PT two minutes, that means I’m waiting for a resource signal for two minutes and when I receive it, I’ll be happy. You can also specify your counts. If you wanted to, you could say count two, but that would make no sense because we’re not sending two signals. That is if you create multiple easy two instances for us, it could be count one because we’re just waiting on one instance. So I’ll just leave it like this. So this sample weight condition is waiting for one instance to signal something within two minutes. And that signal should be obviously positive.

And we know that the signal has been sent to this through the command right here. It says resource Sample weight condition. The rest is the same. The SSH Security group has the same rules. And so the only thing we’ve added in this stack really is the weight condition and the CFN signal script in our user data. So let’s have a look at how this works. So we are in confirmation. I’ll create a stack, upload a template file and this time I will upload Cfnsignal YAML. Click on Next and I will just say CFN Signal Example and I’ll just remove the typo because that’s pretty bad. Okay, now the SSH key is going to be the same as before. Click on Next and on the bottom, click on Create Stack. So now the cool thing though is that I will not pause the video because I want to show you exactly how things happen in real time. So we’re waiting for things to happen. The SSH Security Group is in great progress and the sample rate condition is in Create Progress too.

Now we’re just waiting for our EC Two instance to start up. So my instance is now in Create in Progress. So if I go to my EC Two management console, I should be seeing my EC Two instance that is starting right now. But as you can see, my wait condition is still in Create in Progress. So it’s still waiting. So I’ll refresh again and wait for it to happen. Okay, so my EC Two instance is now running and so if I go back to confirmation, this is usually when things would end. It would say my instance create complete. So it’s going to show this in a second. Create Complete. Here we go. My instance. Create complete. And so the stack is still not completed because the sample wait condition is waiting. So if we go back to stacks, you see that still Create in progress.

So it’s because the wait condition is waiting for a signal. And so this will not happen up until the EC two instance will have configured itself and use the CFN signal. So I’m just going to wait again a little bit for the CFN signal to arrive. And here we go. The sample weight condition has received a success signal with this unique ID. And so that was about 20 seconds after my EC two instance has created and because it received a proper success signal, then we went into a create complete stage. And so that’s the cool thing. The wait condition had to wait 20 seconds for it to complete. Now, if I go to my ECU management console and open up my public IP, I will know for sure that it has worked because the CFN signal sent a positive signal to confirmation.

And so that’s a really cool thing. Now with weight condition we are able to basically say wait a little bit until you can get say okay, your stack has worked because this easy to instance will be able to send a signal right away in the next lecture. We’ll see what happens when things fail because that’s a very important test case for the examination. But that’s it for now. We’ve just seen that a wait condition allows us to wait to complete before we receive a signal. As usual, I will go ahead and delete my stack and I will see you in the next lecture.