SPLK-1003 Splunk Enterprise Certified Admin – Installation and Configuration of Splunk Components Part 4

21. Configure Search head From Splunk Web

Once we have successfully configured our indexes, the next component of Splunk is searched. Since indexer is the core component which starts receiving of the logs, which stores logs and everything, the next component would be searched or UAV folder or Universal folder. You can start with any other component, but for this tutorial we’ll be starting with configuration of searcher. There are total of three different ways where you can configure Splunk searcher. We will see them one by one and you can choose and pick it up whichever you feel is comfortable. But in large deployments, keep the configuration centralized. I would always recommend to edit the configuration files and understand what is changing. When you click or modify any changes using Web or CLI. Let’s see all the three methods that is Web, Splunk, CLI and editing configuration files. Before we begin, let me log into Searcher so that we can start our configuration running on Https on port 8000.

Once you log into the Splunk, the Web component is the easiest part. Probably you can start with modifying all the configuration using Web. But since I’ll be going through other two methods also, it’s good to understand which configuration gets reflected or which file is holding the configuration that you are modifying using web. So that said, we’ll start configuring a searcher or the component of our Splunk to act as searcher. To configure Splunk Enterprise instance as a searcher, click on Settings. Go to distributed search under distributed environment.

Once this page loads up, click on Add new under search peers. To add an indexer, you need to mention the server name or the IP address. We’ll grab our indexer IP address. So this is our indexer. We have not customized any port, so it is 80 89 and the username and password because when you authenticate when Splunk authenticates with the indexer, it will exchange its public keys so that it can log in and run the searches and fetch those results throughout our Splunk environment. The password enter as you can see, we have successfully configured a Splunk instance to act as searched.

The Splunk instance name is by default the host name. We can configure them as indexes. We’ll see at a later stage how we can rename Splunk instance name for now, to validate, go to your search and reporting app that we are successfully set up. And also when you set up, we get a couple of screens. So one is our Uri, the second one is the host name. This gives us a quick picture that our configuration was successful. Replication should be successful. In a matter of time it will turn successful. Similarly, if this belongs to any cluster label no as of now health status? It seems healthy. And any Check failures in recent time? None. And the status of this indexer is enabled. If you want we can disable it.

Similarly, we can delete or current in the actions once as soon as we see status up and it means that it’s successful. To verify, go to search and reporting app to verify the best option would be to search for internal logs. Let me skip the two. So this is our search bar. We know couple of basic searches that we have performed in our early part of tutorial and we have gone through all these menus and functionalities so I’ll directly jump into writing a query. It will be a simple query. Index is equal to internal. I’ll check for last 15 minutes because within 15 minutes we have configured our indexer we’ve got like 5000 events. As you can see there are two hosts in our internal index one is our searches, the other one is our indexer. To better understand these values we can check the host name or we can replace the host name on our index.

22. Configure Search head From Splunk CLI

Here you see the 239 14 dot 239 is our indexer 14 dot 239. Now our search ed which is twelve dot 76 is able to search the logs on the indexer also which shows our configuration is successful. We have configured a Splunk instance to act as a searcher. Now we have done this part using Splunk web. We’ll see how we can do this using Splunk CLI. This is our indexer. We will not be touching any indexer because we’re configuring the searcher. So on searcher I’m logging in. Let me jump to my application user that is Splunk. Yes. Now I’ll go to my Splunk utility bin Splunk before we add our indexer since we have only one indexer as of now, I’ll go ahead and delete our present indexer. Go to distributed search, search peers and delete here. The tricky part is if you want to add indexer using the CLI, you need to mention the username and password using the CLI splunk add search server and the IP what was our index IP? So this is our index IP and the port is 80 89 and you need to mention the remote user name which we mentioned as part of for web configuration the username is admin. Similarly, we need to mention the remote username here.

So now we have mentioned remote username. It is remote password. So this is the syntax. That is you need to have capital P and capital u for remote username and password and search server is the one which is representing this is my indexer add it to my Splunk searcher. So it will ask for my searcher must be informed parameter. Okay, I have not mentioned my password. That is the reason my password is this is my index or password. It is asking for my searched username and password because I’m modifying my searched configuration. It has to authenticate the password is say now it says peer added. Let’s go back here, refresh this as we can see our instance or the indexer came back up again. This is the way we add our indexer to the searcher to configure an instance to act as a searcher. Let us see, we have since already added what happens when we edit the configuration. How to configure searched using indexer by editing configuration. This is quite complex. We’ll go through it slowly.

23. Configure Search head From editing Configuration Files

In the previous video we have seen how to configure Splunk instance to act as searcher using web console. That is by adding indexer IP and poor details that is 80 89 and the indexer IP and using CLI mentioning the passwords of remote user and Splunk searched user. Now we need to see what exactly happens during the search at configuration. That is initially when we enter this details so that we’ll be able to understand by configuring a searcher using editing configuration method. This is a searcher. I have deleted the previously added indexes. Let me refresh to confirm search peers and there is nothing as of now I’ve cleaned off other configuration which were as part of effect of our web configuration and CLI configuration in order to start fresh.

Now we are in our Splunk searcher instance. Before editing configuration here, we need to make sure the certificate that is present in etc from Splunk home Auth and this search. This is the dist server. Keys are the one which is used to communicate between searched and your indexer. So we need to copy this trusted key of searcher to our indexer machine to which location to the same location. Opt splunk etc. Author keys. But make sure you create a directory that is named after your search at host name so that it can identify which key belongs to which server. So I am creating a directory based on the host name. I’ll log into the directory. Now I’ll copy this file probably easy way without displaying the trusted key. No matter, by the time you guys see these videos, I’ll be disabling these machines. You are probably deleting them. So no issues. We need trusted key. Let me create it trusted pam and copy paste the same key. Hopefully there should not be any syntax error. We’ll see that if we face any issues. So this is our searcher. Once we have copied our keys into our search indexer, the searcher keys have been successfully copied. As you can see. This will be our complete location on the indexer where the keys are located. It will be under this server key along with the host name folder where the key is presently created. This is our key. Opt Splunk the configuration file for your search at is etc system localist search. Here we need to place this syntax which is distributed. Search is the configuration stanza name and service is the list of your indexers. Here you can specify, comma and mention any number of indexers you have so that the searcher is configured to run on all these indexes. Let me copy this. Paste it here. Save it. Since we have edited configuration, let us reset this instance. Once it is up, we should be able to see without entering any credentials.

But with just copying our keys and we know that configuration resides now on dist search cons. Now we’ll be able to see our indexer should be up and running. Indexer logs should be able to fetch from our search and as you can see the state is up, replication is successful and the health status is healthy and this will be our index app. To validate that we can quickly run a local internal splunk log search index is equal to underscore internal. Let’s run for last five minute window that is real time search. You’ll see there are two hosts 14 239 is our indexer. Now, we have successfully configured our searcher using three different methods to conclude one using setting that is splunk web using distributed search. The second one is by adding search server using the syntax opt splunk bin search iPhone server followed by the remote username and password along with IP and port details. Post entering this command it will ask for your splunk search and login. So once we have done that, we’ll be able to see similar screen where the indexes is up and able to communicate with your searcher. Running an internal index search should be able to give you that your indexer is communicating with searcher and your splunk component is officially searched now.

24. Configure Heavy Forwarder using Splunk Web and CLI

Now since we have seen already how can we configure a Splunk component as a searcher and also how to configure Splunk component as an indexer. Let us go through how we can configure our heavy forwarder. That is making an instance to receive data and forward it without storing any of those data. The evolveder functionality we already know from our previous modules or our previous discussions on avoider that it passes the data that it is received from the universal forwarder and it sends it to the index cells. We need to do a similar step of adding and receiver on specific port that we have seen three ways of doing it in our indexer configurations. Now for adding the receiver a specific port I’ll be using a simple method that is Splunk web console. This is our heavy forwarder instance. That is 8000 port. So probably we should be able to do it with our CLI itself. Let me log in. Let me log in to my application account. That is splunk. Okay now. Opt splunk. We all know by now that how to configure receiver using Splunk CLI. Let me run the command enable.

Listen this step we have performed the same in configuring indexer. We are enabling listener of Splunk for port triple nine seven. It is asking for my Splunk credentials. It says listening for Splunk data. That is Splunk data on port triple nine seven. We are configured receiver. Now we can go ahead and check in our web settings forwarding and receiving. It should be under receiver. This is our AB forwarder instance. We have successfully set up the app functionality of receiving the logs. Now we know the other two methods. If you have still difficulty in figuring out the other two methods probably you can revisit the indexer configuration video. From there you’ll be able to see what are the other two methods that is the web and directly editing the configuration files. Now we’ll be moving to creating a forwarder like forwarding the data that has been received from universal forwarder and pass it and forward to the external which is the second functionality of our every forwarder. For configuring forwarder we have again three different methods. We’ll be going through them one by one. The first method is of course by Splunk Web or the GUI. Go to settings forwarder and Dressing. You will be revisiting the same page. Click on Configure forwarding or click on add new. You’ll go through Configure forwarding so that we will see there is nothing before we’ll click on new and enter the host and the port where we need to send the logs. We know the port that is triple nine seven. The host is our indexer. Let us grab the indexer. The port is triple nine seven. We know because during our indexer configuration we have enabled receiving on triple nine seven. This will be our forwarding port.

Click on save. So this should give the functionality of every forwarder to do the job of initial parsing and sending it to your indexer. There are other functionalities like routing of events masking your data, filtering your data which will be looking at the later part of our tutorials. For our configuration perspective, this is how we place an AV forwarda so that it receives the logs and it sends it to our indexer. To understand our second method, we’ll just delete this configuration. We’ll re add the same configuration using second method. That is our CLI. Now we have deleted the configuration log into our Splunk AB forwarder instance. Go to Opt Splunk utility. The command would be for adding indexer in the previous tutorial while learning configuring searches. We used search server. Here it will be forward server. That is it is forwarding the logs to a server that is your indexer on port 97. So it says added forwarding to this IP on this port. Let us refresh this page. We should be able to see those codes. Configuration yes, the configuration has appeared from R splunk CLI. Let’s see how we can do this using editing the configuration files.

25. Configure Heavy Forwarder using Splunk Configuration File Edit

So now we know about how to configure AV forwarder like forwarding of data to the indexer using web and CLI. Now let us configure it using editing of configuration file. If you log in to our AV forwarder this is our AV forwarder machine. Previously we added let us quickly check for editing configuration file perspective. I will just remove those configuration so we can read our indexer IP and the port details using configuration editing as we can check now we are not forwarding data to any of the index service. Let us go back to our AV forwarder. Yes, this is our AV forwarder to forward data out of Splunk to any instance let’s say from unites forwarder to indexer or units forwarder to AV forwarder or AV forwarder to indexer or for simplicity, if you want to send the data out of your Splunk instance, you will be always using this configuration file.

That is the configuration file name is Outputs conf. So these are named very specific to their functions that are carried on the configuration files. When you say Inputs conf it is used to get data inside Splunk. When you say Outputs conf it is used to send the data out of that Splunk instance. There is typical syntax to follow. This will be the syntax that we need to edit in our outputs conf. That is create a TCP output configuration stanza and mention a default group of indexer or your AV forwarder. That is this is our default AV forwarder. So this will be our default AV forwarder and mention what are the members in that group. For our group we have only one indexer. If we have multiple indexure, we can add them by placing commas any number of EV four orders or indexers to where we need to send the data out of Splunk. This is our TCP out server. That is the output where Splunk AV forwarder is sending the passed output. I’ll copy paste this save this file.

But we’ll still not be able to see the configuration because we have edited one of the configuration file which requires restart. So let’s proceed with it restarting Splunk. Once you restart Splunk, you should be able to see the new forward server that has been added which is the default group that is default AV forwarder group name and we have one server in it. First blank has started up. Let us check. I’ve entered my password wrong. Once it loads up, you’ll be able to see the new forward server that has been added. This is our new indexer. As we understand the AV forwarder we have done two configurations. One is forwarding, one is receiving. Receiving is similar to the step that we have performed for the indexer. The forwarding part is similar to the universal forwarder which we’ll be seeing in sometime. That how you can send the logs from universal forwarder to your index.