SPLK-1003 Splunk Enterprise Certified Admin – Installation and Configuration of Splunk Components

6. Testing Configuration Precedence

The easiest way probably is to check is by searching for our internal logs. Index is equal to internal. Let us run for last 15 minutes should be fine as you can see in last 15 minutes there are two host names. This was before we were testing hierarchical. If we keep it for just like last five minutes we’ll be able to see our configuration from host. System local has been picked up as per our configuration. This is right. So this overrides any configuration that has been defined in this location? We didn’t change any default location because it is highly recommended to edit any configuration under system default. We have edited these three configuration out. Of these total, the system local got the highest preference. The configuration, as you can see, as reflected in our host name.

Now, what happens if I eliminate my first one? By understanding it should be clear that the host name should be picked up from system local. Let us remove our configuration from system local etc system local this is where we defined our configuration. Let us remove this or you can comment it out or you can completely remove it. I’ll go ahead and restart my Splunk instance. What do we expect to have on a host field now?

It should be host under app local so that our second preference should be picked up from application local directory. Our splunk has successfully restarted. Let me login let me rerun the search for last five minutes. As we can see now there is a new host entry that is host under app local. Since in the last five minutes. This was the default system default before editing any configuration and this was after editing or specifying the same configuration under four different hierarchy, the system local clearly won and we saw the first one system local reflected when we see the second one.

When we remove the configuration from here, even though the default is there, it will be overwritten by our app local. It picked up our second hierarchy as per our understanding. Now let us go ahead and remove our app local also so for that we’ll be going under etc apps. This is the app name where we edited the configuration. We will remove the local configuration now the final fight is between to pick up the configuration. The final fight will be from app default and system default. Let us restart a Splunk instance. Once we have restarted our Splunk instance we should be able to see our latest host entry that will be host under app default.

7. Concluding Configuration Precedence

Now our splunk has restarted. Let us log in. I’ll search for last 1 minute so that we’ll get just the latest events. As you can see now we have a new entry host under app default. Our understanding we are clear that when the same configuration is defined in all four locations, whatever is defined in System local will come up as a winner and Splunk while starting up picks up any configuration here as it’s the final configuration. If it couldn’t find the configuration here, it checks for these three directories. If these three directories, App local will be the winner and it will have the final configuration while starting the Splunk. Similarly the app default and system default when it has conflicting or the same configuration, the app default will have the highest priority for any configuration between these two. If Splunk while starting up it couldn’t find any configuration or customization that are defined in these three, it will look for our system default.

Let us go back and remove our default configuration from the app default directory. I’ll comment out these restart my Splunk instance so that now we should be back to normal. We have not customized any configuration whatsoever. It should be picking up directly from the system default location splunk has started. Let me rerun the search. As we rerun the search, if you check for the latest event, it will be our default host name.

System Default if you want to know where the system default the host name it is picking up is it is the system default inputs. It is mentioned as Decide on startup. So if you have capital host name command defined on your OS, it can pick up from the OS. So what this Decide on startup does is while starting up the Splunk, it will check for the host name of your machine where Splunk is installed and it will take that host name and it will assign it to your logs that are generated out of those machines. To be very clear, when you are troubleshooting some configuration or you are editing some configuration in app default or app local you see that it is not reflecting upon syntax being right and everything.

But there might be a configuration in System Local which might be overwriting whatever you define under these three locations. And also always keep in mind never ever try to edit this default location folder. Let me show you. By default, even though your system account is used to run Splunk privileged or normal, these files in System default will have only read permission. As you can see, this is the system default and all as read permission. Splunk highly recommends not to edit these files so that if you mess of any configuration your Splunk might never start. Make sure you never touch these files if you want to edit them. Copy these files into any of these three locations and modify them.

8. Installation of Splunk Enterprise

This tutorial I have created four machines to understand how we are going to install Splunk Indexer, Splunk Searcher, Splunk Heavy Forwarder and Splunk Deployment Server which we will be also using a license master to go to this tutorial I have configured all the credentials and created our application. Users and met are all prerequisites of Se, Linux disabling THP Disabling, Firewall Rules all these have been taken care so that we can get right into our installation part. If you are not clear about the prerequisites, just go back a couple of tutorials where we have discussed exclusively on the prerequisites of our Splunk installation. Let me log into one of the Splunk instance. This is a splunk searcher. I’ve logged in by default as easy to user. I’ll switch into privileged user. So this is the command use it in Linux to switch into privileged user.

Now I am root as you can see here and this is our Splunk searcher. I’ve already downloaded the Splunk installation package which is of the latest six two. Now let me show you how easy it is to install any package in Linux or how we are going to install our Splunk. All I’m doing is it’s rpm reedit package Manager if an I for install iPhone b for verbose mode and if one h for human readable output I’ll mention the filename that we are going to install. That is our Splunk enterprise package six to Enter. As it progresses we’ll see the installation is almost done. You can consider since we have installed this package on a machine that was referred by us still Splunk doesn’t know that it is a search and we need to configure that.

As of now we can consider one instance of Splunk is installed. Let us go to our next component. I’ll copy the same command so that it will become easy for installation. I logged in as privileged user. Now let me check whether I have the package. Yes, I have the package downloaded here. So the same command I copy pasted it enter children without any issues. That’s it. We have installed our Splunk instance on the indexer. Now this is our AV forwarder. I logged in as a normal user. Let me switch to privileged user quickly verify whether we have the installation package. Paste our command. Hit enter. That is it. In a matter of minutes we have installed three instance of Splunk. We have one more left. That is a Splunk Deployment Server or License Manager. Switch to Privilege User, verify the package and paste the command hit Enter. You can automate it by writing a small script like Adscript and provide all the IP addresses where you want to install Splunk components. This should be the basics so that one script should be able to execute everything. Now we have installed four installations instances of Splunk. Let us check out. These are splunk full instances. How to install Splunk universal forward app.

9. Installation of Splunk Universal Forwarder

In our previous tutorial we saw how to install Splunk on Indexer heavy forwarder deployment server and searcher. For this tutorial we’ll be using our local machine that is my laptop as a remote agent to the indexer in our cloud. This is the Splunk forwarder package which is of latest 662. The steps will be similar on any Windows platform. Just check this box so that we are accepting license and there is a customized option to change the default Splunk installation directory. We have also seen the default Splunk home when we are going through the directory structure of Splunk. This is your default Splunk home. If you are installing full Splunk instance, it will be C program file Splunk.

For this tutorial we’ll be showing a demo of Splunk Universal Forward Air installation which is similar to as Splunk Enterprise. So I’ll keep this default setting as it is and I’ll be clicking next the password it is asking here is for the SSL certificate. This password if we have SSL certificate like when we are hosting or sending it to cloud, we can upload it here. Or if we are using default Splunk generated certificate, we can leave this blank. I’ll be running using local system account. So what all we need to wait let me enable everything so that we get most of the information to our Splunk instance. And if you have any custom directory, let’s say D or E file systems where you need to monitor, you can specify it in this path.

And also if you are installing on Active Directory service, make sure you check enable Active Directory monitoring. This is one of the important configuration. Like if you have a deployment server in your environment, you can mention the IP and hostname during the installation. We’ll come to this part when we are configuring our deployment server how to add this configuration as part of the installation or as part of using Splunk CLI or using configuration files. As of now, leave this blank. Continue similarly now it is asking for indexer. Even indexer IP address will be coming to this part. When we are configuring how to set up an indexer. Then we will be updating this configuration in your universal forwarder and we’ll be showing three methods splunk CLI editing configuration and we have completed all this installation. But still those instances have not started up. We’re going to configure them one by one and start those instances. So let this installation finish and we should be able to proceed with our configuration of this installation.

10. Installation of Splunk Search Head

In our previous two lectures we went through how to install Splunk on Linux as such ed index, avoider deployment server and how to install on Windows. We looked out for only universal forwarder that we have installed in our local laptop and these will be sending logs to our AV forwarder. Then avoider will pass the logs and send it to our indexer. Now let’s see some of the basic comments for the everyday operation as Splunk go to your Splunk installation directory that is your Splunk home c program files splunk universal forwarder go to Bin. There should be Splunk exe stop will let you stop the instance. Similarly start or restart option should be able to bring up your service. Since our Splunk universal forwarder doesn’t have a web GUI content, there is only one port that has been used, that is 80 89 and rest of the ports are not being used. The universal forwarder does only one job of fetching the data and forwarding it to other Splunk instance.

Now we know how to start, stop or restart our Splunk instance in our windows. Let us see on Linux couple of tutorials back we install our Splunk instance. Now by default Splunk is installed under opt Splunk directory. From the prerequisite that we have gone through earlier tutorials, we know that it’s always recommended to run Splunk as a non root user. So for that purpose I’ve created a user named Splunk. I’ll be using this user to perform all Splunk actions like starting, stopping, editing configuration. Any related Splunk task will be carried on under this user. Now I’ve changed my user to Splunk. Let me start Splunk for the first time. You’ll get a couple of screens which I’ll go through them one by one. As soon as I hit opt Splunk start, it popped up in license agreement where in Windows case we had a checkbox just to check the agreement.

Here it displays if you want to read just it on space so that it will continue showing the entire license. We are not bothered about the license at this moment. So I’ll just quit pressing Q and then hit Y to accept the license. Then enter. Now our Splunk search had been started successfully. Here there are a couple of messages. Let us go through one by one. Here is our license accepting. This was the last line before accepting license. So once we have accepted, it says this is the first time you are running the Splunk on this machine. Yes, we have just installed and we have just started up. It is just copying some of the configuration from default to local. We’ll go through them one by one and it generated certificates. Those are Splunk internal certificates for communication and exchange of data and even Https it generates these certificates. Here is the certificate that it was generated.