Google Professional Data Engineer – VPCs and Interconnecting Networks part 4

7. Lab: Working with Firewalls

In this demo we will be studying Firewalls and we will see how their configuration can affect communication between instances in a network. Let us start off by creating a new virtual network. So we navigate to the Vpc network section and click to create a new Vpc network which we call Learn Vpc. We choose to create our subnets manually in custom mode. Our first subnet is for our web server which we create in the US East region. We enter a cider block. Our second subnet is for our databases, also in the US East region. And finally we create a subnet for our service. Let us now go ahead and create our instances.

So we navigate to the Compute Engine and VM instances section, and the first instance we create is our web server. To this we attach a network tag to label it as a web server and we want this to be created in our new custom Vpc network and in the web server subnet. While this is being spawned, let us just go ahead and create our database instance. This will just have a different network tag and it’s going to be in the database subnet of the same Vpc network. We can now go ahead and create our application service. Again, we attach a different tag to this and it’s going to be in the subnet corresponding to the service.

With our instances ready, let us navigate to the Firewall rules where we will see that the default Vpc network already has a set of pre configured firewall rules to allow Ssh, RDP and Icmp access to these instances from anywhere, as well as all other access from within the Google Cloud network. All of these though are incoming rules and we will see that there are no outgoing rules specified because outgoing connections are unrestricted. Let us now create our own firewall rule which we will call Learn Firewall One. We attach it to our custom Vpc network and note that there is a priority of 1000 which we will leave untouched for now.

We want this rule to apply to all instances within the network and we don’t want to restrict the rule to any specific source. We want to restrict access only to certain ports and protocols. So we specify the Ssh and Http ports of 22 and 80 along with an extra port of 33 or six. Once our firewall rule has been created, let us navigate to our web server instance and Ssh into it. And from over here, let us try to ping the database instance. Let us debug this connectivity issue by taking a look at our firewall. Once we navigate to it, we see that the Icmp protocol which is used by things is not mentioned in this list of approved protocols in this firewall rule. So let us just go and edit it to include Icmp.

Once we go in, we just add Icmp at the end and hit Save and we just navigate back to our instance. And this time the print to the database works. Let us now quickly navigate to our application service instance via Ssh and we will try pinging the database from here. And we see that this is also successful. So the database instance is reachable on ping from both the web server as well as the application service. Let us now go ahead and create another firewall rule to prevent the web server from directly accessing the database. So we create a new rule which we shall call Learn Firewall II. We want this to apply to our custom Vpc network, and we want this rule to supersede the one previously created.

So we attach a higher priority of 900. We want this rule to block access to the database. So we set the action to deny. Since we are blocking connections where the database is the destination, let us specify the target tag as DB which is attached to our database instances. And finally, we want to block connections coming in from the web server. So the source at the range this rule will apply to is the one corresponding to the subnet in which our web server lies. And finally, the protocols and ports which we wish to block our icmp for ping and port three three six as well, which is often used by MySQL database. Once this firewall rule is ready, we can clearly see that this rule applies to any connections coming in from the web server subnet.

Let us now move over to the web server instance and see if the ping to the database is still working. And we can see here that it has been blocked. Just as a firewall rule was set to do. Let us now confirm that a firewall rule was only blocking access from the web server and that ping from the application service instance are still possible. So we navigate to that. And yes, the ping is successful from here. So this means that a firewall was configured correctly. As a final step in this lab, just as a best practice, we’ll just go clean up all the resources which were created as part of this training. So we first clean up our instances, and once that is done, we clean up the firewalls. And finally we just delete our custom Vpc network. All right, that concludes this lab. I hope you found it useful. Thank you. Bye.

8. Lab: Working with Auto Mode and Custom Mode Networks

In this demo, we will take a look at the various virtual networks available in Gcp. We will create some Firewall rules and provision a number of instances across the network. We will also explore how a combination of Firewall rules and the placement of instances across various networks affects communication between them. Let’s dig right in. By exploring the default network, Gcp creates automatically for each project. Virtual networks, also known as Vpc networks, can be found in the console under the networking section. As you can see, the default network contains a subnet for each Gcp region. Moving on now to routes.

And we will see that Gcp creates routes automatically for each Vpc, one for each subnet within the Vpc, plus an additional one for an Internet Gateway. Now that we have taken a look at the default Vpc network and Gcp, let’s move along to an automatic network which is best understood by creating one. So, once we begin creating the new network, we specify a name, a description, and once we specify the creation mode as automatic, we will see that a number of subnets in specific regions will be created automatically. There are also a number of pre configured Firewall rules which one can include in their automatic Vpc network. These include rules to Ssh or RDP into instances which are provisioned on the network as well as the ability to ping them.

Additionally, there are two Firewall rules which are implicit to every Vpc network. One of these is to deny all incoming connections to instances on the network, the other one being to allow all outgoing connections from these instances. If you would like to automate the provisioning of this Vpc network, Google provides this Nifty tool where you can see the exact rest call which you would need to make based on the parameters you have provided in the console. You could also take a look at the specific shell commands which are required in order to provision the Vpc network as specified in this demo.

However, we will just be creating this Vpc from the console. Once we hit create, we just wait for the Vpc to be provisioned.

We wait, we wait. And yes, as you can see, the learn auto Vpc has been created with one subnet created in every Gcp region. We then proceed to see if the routes have been created as expected. We may need to refresh the page in order for the routes to be loaded onto the UI, but once it does load, we see that there is one route created automatically for each subnet in this newly provisioned Vpc network, plus there is a default Internet Gateway route as well. Upon closer examination of the subnets and routes in our newly provisioned Vpc network, we observed that these are identical to the ones in the default BPC network moving along to the Firewall rules.

And we observe here that the four firewall rules which we selected when creating the new Vpc network have been provisioned as well. Now that we have an idea of what an automotive Vpc network is, let us create a custom virtual network. To do this, we click to create a VPC network. We also supply a name and description, as we did before. But for a custom network, we do have to specify details for each subnet which we wish to include. These details include the name for the subnet, the region in which it will lie, as well as a Cider block to denote the IP address range for the subnet. In our example, we are adding three different subnets in two different regions to this network.

When adding multiple subnets to a network, we need to be careful that the IP address range which we specify for each do not overlap with each other. This is something which we did not need to worry about in an auto mode network. Once the subnet information has been filled in, we just hit Create and then we wait for the Vpc network to be provisioned. We wait. We wait, and once it is ready, we do observe that it has three subnets as we had specified. We then proceed to see if the routes have been created as expected. So we expect to see four routes, one for each subnet, plus one for an Internet gateway. And yes, we see that those have been created as expected. Let’s proceed now and check to see if firewall rules have been created for this network.

And we noticed that for a custom Vpc network, there are no firewall rules generated automatically. So let’s go ahead and create one on our own. So with this firewall rule, we will allow Ssh, RDP and Icmp access to any instance which is provisioned in the network. We select the network to which this firewall rule will apply to our custom Vpc network, we leave the priority as is, and for the direction of traffic we leave it at Ingress, which implies that this firewall rule applies to any incoming connection to instances in our network. We use this feature called Target Tags, which implies that this firewall rule will apply to any instance on the network which possesses this tag.

We do not want this firewall rule to filter out incoming connections based on the origin, so we specify the source IP range as such. However, we do want to allow only things Ssh and RDP connections to enter the network, so we list these down in the protocols and port section. Once this is done, we just hit Create and wait for the firewall rule to be provisioned. Now that we have created our own custom Vpc network and also wrote a firewall rule for it, let’s try to edit this network by adding a new subnet. So we click on the network, hit Edit and add subnet, and we try to create a new subnet in the Europe West region, but we specify a Cider block which overlaps it’s one of the existing subnets in this network and we see that Gcp will not allow the subnet to be added.

So we cancel this subnet addition to our Vpc network and move along to provisioning instances on all the Vpc networks. We have to start with let’s provision an instance in the default network. So we create this in the US East region. We select our Machine type and we go into the Networking section to see that the default network has been automatically selected. So here done. And then create. And we just wait for the instance to be provisioned. We then move along to the second instance. This instance we will create in the Auto Mode Virtual Network which we have provisioned. We will create this in the US Eastern. Again we will select our Machine type and in the Networking section we will have to change the default value from the default network to our own Learn Auto network.

Once this is done we just hit Create and wait for the instance to be provisioned. Moving along now to instance number three, which we will provision in our Custom Mode Vpc network. So the details for this are very much the same as in the previous two cases. So yes, in the US East same machine type, but this time in the Networking section we will select our Learn Custom network. We will provision this in the subnet one A and then hit Done and Create. Moving along now to instance number four, which is pretty much exactly the same as instance number three. In this case we will just provision it in a different zone. You choose US East One C in this case but everything else remains exactly the same.

The same machine type, same network as well Learn Custom and the same subnet as instance number three. Once this is done we just create and move along. Now to instance number five, our last one. This instance will also be in the Learn Custom Vpc network. However, in this case let us provision it in a different zone. Let’s select one from the US West region and then we change the network to Learn Custom. And this time we select subnet Two for our instance. Once that is done, again we just hit Create and wait for the instance to be provisioned. Now that we have all our instances, let us Ssh into the Learn One instance which you will recall was provisioned in the default Vpc network.

That network also had an Autogenerated Firewall rule which allows Ssh access into the network. Once we have given our browser the necessary permissions, we are able to bring up the terminal and let us try to ping Learn One itself using the host name. We see that this ping is successful which shows that the DNS translation works for Learn One. Now when we try to print Learn Two using the host name and remember, Learn Two is on the Learn Auto Vpc network. This fails because DNS is scoped to the network. However, let’s try to ping Learn Two using its internal IP address. But we notice here that the Learn Two IP address is the same as that of Learn One because they’re on different networks.

So this ping will work and let us now try using Learn Two’s external IP address and we should see that this also works. Let us now try to ping the Learn Three instance, which is on our custom Vpc network using its external IP address. When you try this, we see that this does not work. Usually when there is a connectivity issue like this, it makes sense to first point the finger at the Firewall rules. So let us navigate to the Firewalls page and see if there is anything which we have missed when we created the Firewall rule for the custom networks, a quick glance reveals that the Firewall rules which were autogenerated for the Auto mode and default Vpc networks apply to all instances in the network.

However, our custom Firewall Rule only applies to instances with the Allow Default Network tag associated with them. So let us go add this network tag to our instances in the custom Vpc network. We go over first to the Learn Three instance, we hit Edit and we see that we can add a network tag. So we specify Allow Defaults, which is what was stated in our Firewall Rule. And once we hit Save, let us head over to our terminal window on Learn One, and this time the ping works. Let us now Ssh into the Learn Three instance and let us try to ping Learn Four, which you will recall, is not only on the same network, but on the same subnet. But this fails for some reason. Let us ping Learn Five and see what happens.

And this fails as well. To debug the problem a little further, let us try to install Trace Route and see where along the path to these instances the connectivity fails. So once Trace Route is installed, we try to connect to Learn Five and we see that it fails on the first hop. Let us investigate further by connecting to the Learn for instance using Ssh. And when we do so, we notice that the Ssh connection cannot be established. And if you remember, we had added the Allowed default tag to the Learn Three instance, but not to Learn Four and Learn Five. So we could either add the tags to the instances or I’ve decided that the Firewall Rule is a little too restrictive.

So let us go edit it so that the Firewall Rule applies to all instances in the network rather than just the ones with the Allow Default Network tag. So we go over to a Firewall Rule and hit Edit. And for the targets we specify that this applies to all hosts in the network. And once we hit Save, we wait for the Firewall rule to take effect. And then we head over to our Learn, for instance. And this time the Ssh connection works. Let us now check the hostname translation by Pinging Learn for itself and we see that this works. Let us now check Learn Five, and this is no longer a problem for us. So it looks like the changes to the Firewall have resolved our connectivity issues. Just to be sure, let us also check Trace route.

So we check the part to Learn Five from Learn Four, and we see that this is done on one hop because they lie on the same network. Now that we are familiar with Firewalls, let us explore why we might consider converting an Auto Mode Vpc network into a custom one. So let us take a look at the Auto mode network which we provisioned, and let’s just say we wanted to add a subnet, or remove one, or even modify an existing subnet to include a larger IP address space. And we see that in the Learn Auto network this is just not possible. So let us modify this and change it into a custom mode network, which we can do easily enough from the Vpc Networks page by clicking on the mode and changing it to custom. Once this is complete, we will see that there are many more options available to us.

We can navigate to the network, and we now have the ability to add and remove subnets from the UI itself. Let us start off by deleting one of the subnets, specifically the US central subnet. So we check that and we hit delete. And once this is done, let us try adding another subnet to this network. So we click on Add Subnet and create this new one in the US East region. But unlike the other subnets in this network, which have a slash 20 cyber block, this one will have a slash 26, which means there are fewer IP addresses in the subnet as compared to the other ones on the network. We will then go on to expand the subnet. So once the subnet has been added to the network, which will just take a couple of seconds, we can now proceed to expand it.

However, you’ll notice that there is no way to do so from the console at least out of this recording. However, it is possible to expand the subnet from the command line using the G cloud shell. So let us try that out. Once we bring up the shell, this is the command which we can use to expand the subnet. So the name of the subnet is New US East. And once we execute this command, we’re going to expand it from a 26 to a 23 cyber block, and we just say yes. And once this is complete, we just move over to the console to see if this is reflected in the UI. So we navigate to the UI to our network. We will need to hit refresh in order for our changes to be seen. And yes, once we do that, we do see that our command executed from the command line has run successfully, and we now have an expanded subnet.

So that concludes this demo on GCP’s virtual networks. However, before we go, it’s always best practice to just delete all the resources which are provisioned during this exercise, just so that we don’t inadvertently incur massive hosting fees from Gcp. So let us start off by getting rid of all the instances which we created. These are the five instances. So we just select all and hit delete. And once these have been terminated, we can just go over and delete all the virtual networks which were created as part of this exercise. So we do have to go into each network in order to delete them. Once the Learn Custom Network has gone, we just proceed to delete the Learn an Auto Network. And once that is complete, we have completed this exercise. I hope you found this useful. Thank you.