SPLK-1003 Splunk Enterprise Certified Admin – Installation and Configuration of Splunk Components Part 3

16. Configuring Indexer: Enable Reciever

Then what is indexing? Indexing is a process of breaking down of events into smaller piece known as parsing and storing of data. So indexing is a process where the component of Splunk which processes the data and stores the data. Now let us see how we can configure indexer in couple of ways. The first step in configuring indexer which is our indexer for my clarity so that I don’t make any mixture, I’ll close all other instances of Splunk and I’ll keep only my indexer screen or the session open. This is my indexer. Let me go to Splunk home. That is Opt Splunk. Now I am in splunk home. Step in configuring an index series to enable receiving of logs. This can be from either AV forwarders or our universal forwarders enterprise instance which is not doing anything.

First step in configuring or making this component as an indexer is to enable receiving of logs similar to any other configuration in Splunk. We have three methods to achieve this. The first method will be the Splunk GUI. Let us log into our Splunk GUI or we can also call it as Splunk web instance which is running under Https. I’ve already logged in so it is directly taking me to homepage. Once we are logged in, click on Settings. You can see forwarding and receiving. Click on forwarding and receiving. Once Forwarding and receiving page loads up, we need to configure risk instance installed act as your receiver. On the configure receiving part, click on Add new.

It takes you to this page where you just need to specify a port number on which you would like to receive the logs by default splunk like by default representation of receiving of logs. On indexer it is triple nine seven. Let us enter the same port and click Save. It says successfully save triple line seven. Now, as we can see, Splunk is listening on triple nine seven port. If triple line seven is already used by some other application, make sure you can change or add a new port to receive on some other, let’s say triple nine eight. You can add any number of ports to make Splunk listen on that specific port. We’ll be using Throat, our tutorial. That is triple line seven. This is one of the ways in which you can add receiving port on your index.

17. Enabling Reciever from CLI and Configuration File Edit

In our previous tutorial, we understood that first step in configuring any index is to enable receiving of logs from AV Forwarder or Universal Forwarder. We have confirmed that we can enable receiving from the Splunk console by using Splunk GUI. Let us see how we can enable receiving of logs or enabling reception on the indexer using Splunk CLI. We’ll be going through all these three but Web CLI and editing configuration. Whichever you feel comfortable, you can pick it up and start using the same either Web CLI or editing configuration file. To enable reception of logs on Splunk, we need to invoke the utility the same utility Splunk which will be used widely for adding, changing or modifying any configuration in Splunk, including removing. We’ll use the Splunk utility with enable listen on port number seven.

So it says Splunk started listening for data on triple nine seven and also it says particularly listening for Splunk data. It doesn’t say for listening for all the data. We will see how to enable listen to all the data in the later part of the tutorial. For now, make sure it is listening only for Splunk data on the triple nine seven port. To verify this, let us refresh our forwarding and receiving. This is before configuration, so nothing is displayed here. Once it reloads, we should be able to see our newly created connection. Yes, this is our triple 97, which we added through CLI rather than Web Console. Now, let me go ahead and delete this. We’ll see how we can add this configuration directly by editing the configuration. To confirm, let me refresh this.

We do not have any receiving data on any of the ports. Let me go back and go to one of the editor opt Splunk etc. Or configuration directory. System Local during the course of this module of the tutorial, we’ll be editing all the configuration in the Local. But it is highly recommended not to enter any configuration in the System Local because we know by the file hierarchy, System Local is at the top so that anything you write here, it will override configuration present anywhere on the Splunk. For this tutorial, we will be using System Local to change the Splunk configuration, but it is highly recommended not to edit any configuration or place the configuration under System Local. We’ll be seeing how to edit the configuration at Enterprise or Organization level when we move to building our own infrastructure for Enterprise level with multi site clustering and high availability in our Amazon cloud. As I explained, this is for part of explanation for the beginners to understand that you can change configuration by editing files. Hence, we’ll be placing the files in System Local. Proceeding under System Local, there will be a file called Inputs Conf which is created by default for taking the host name. We will enter a new stanza. If you don’t know what is the stanza, there is a quick way before going to any documentation. Just go to default.

Not here, just go to default. Directory inputs conf this is basically your documentation which it contains all the default configuration. So I’ll search for Splunk TCP. This is our splunk TCP stanza syntax. Let me see if there is any other. So that is it. We understood. What is the stanza for Splunk listener? I’ll be adding the same. This is for receiving the Splunk logs splunk stanza followed by coler double says triple nine seven this is the configuration for enabling receiving of Splunk data on triple line seven port. So let us restart to see whether we have picked up the configuration. No, because we have directly edited the configuration, give it a restart.

Once we have restarted, we should be able to see our new inputs that are added by editing the configuration file inputs conf splunk is almost up. Yes, it is up. Now our session has expired because of the restart. Let us log in my new password. As we can see now we have our triple line seven port which is configured using editing configuration files. Now you understood there are three different ways of adding receiver on the indexer or taking the first step for configuring a splunk instance as indexer that is by splunk file splunk CLI and directly by editing configuration. Feel comfortable using all the three options because you’ll be able to understand better that when you change a configuration, which file is reflected and where it has been placed.

18. Default Index

Now we have completed our first step in configuring the indexer the first step is to understand what different components or different terminologies that are used as part of indexer configuration and we have configured the first step the second would be to create indexes what is index? Again, it is a group of index we have a Splunk indexer setup ready to accept data but once the data is received it will be stored in the default index when I say default index, it is Splunk. As part of its installation it will have a default index. All the data will go to default index called Main. So by default if you go to settings indexes you’ll be able to see an index named Main. This will be our default index. Anything that starts with underscore is a representation that these indexes are used for internal Splunk application purposes. Die is used for storing your searches. These are some more internal indexes which are used by Splunk itself. The main point here to understand is the default index in Splunk is named as Main. We will see how it works once we start sending the data from our Windows machine that is our local PC where we are accessing the Splunk console to the indexer. So from our local PCs we have collected a lot of Windows configuration during the installation which we will be sending it to our indexes on indexes. The second step is creating the indexes. When we come to indexes we understand that we can create our own index based on technology like Windows, Linux and Mac et cetera.

Throughout this tutorial we will see creating different these kind of index but its complete control is at the will of Splunk admin or architect to create any number of indexes. For example, you can even create index based on the application which are running or the logs are being fetched. From that machine we can create the index named as Apache or IAS or even Database ETCA. Even you’ll be able to create indexes named based upon teams in your organization like Team One, Team Two or Team Three, respectively.

But always create reasonable names because indexes are the best place for doing data segregation on Splunk. For its users to access the reports, dashboards or any visualization that you create let’s say you have two teams Windows and Linux team you can create two indexes named Windows and Linux you can give access for the windows team just to the index windows where they can search throughout the splunk. But they will get results only from windows index. That means their access is restricted to only this data. Similarly, if it is a Linux team give them access only to Linux index, so that even though they try to search throughout this plan, they get results only whatever it is necessary for them. Since we know indexer refers to holding multiple index, let us create some of the index which we discussed earlier like Windows Linux, Team One, Team Two or Apache or even database similar to any other configuration. We can create indexes by three methods. We will see them one by one.

19. Index Creation From Splunk Web and Splunk CLI

The first one. This is our indexer machine. The first one is using Splunk web that is go to settings click on indexes. On the top right you can see new index. Click on new index. Give the name windows. So now, as you can see, there are a lot of optional component you’ll be able to understand further when we come to indexing and how the clustering works and how the data is being stored in splunk to understand. What is HomePath, cold path and third path. For the simplicity of understanding the indexer configuration in this part of the tutorial, we’ll be skipping this and we will come back to this at a later stage. For creating new index, just give the name if you want this to be of huge size make sure you select whatever the option required let’s say 5000 GB so that my index can store up to 5000 GB not necessary, I’ll keep it 100 GB. 100 GB of index for Windows should be more than enough for this tutorial purposes but depending on your organization you can increase it to any limit. It’s totally customizable rest of the options we’ll come back to it as part of later process.

So for creating index and just specify the name specify the size of the index click Save. Here we will be able to see our newly created index windows now we created index windows using Splunk well, let us create our Linux using Splunk CLI this is our Splunk indexer let me check where am I? I am in Splunk home but I have a practice to enter complete command so it is better so that wherever you are, you can run this full command opt splunk pin that is the Splunk utility again the command would be splunk add index our name of the index is Linux. Let me click enter it is asking for again username and password this is your Splunk privilege user so once your password is successfully authenticated you will receive a message that your index has successfully added. Let us see here once you refresh, you’ll be able to see the Linux index which has been created with default parameters similarly, if you want to mention the size you can mention Splunk add index Linux followed by its argument that is size is equal to 200 GB which should create an index with size of 200 GB. For now, we understood how to add an index from the CLI. Now let’s see the third option that is by edit the configuration file all the configuration of creating index, their path, their size and the location of the index is created under a file called Indexes.

20. Index creation from Splunk Edit configuration file

In the previous configuration of indexes we have done a couple of things. One, we have created receiving using three different methods. We have started creating indexing using two different methods that is CLI and Web. Now, the third important method is editing configuration web whenever you try to edit data configuration file, the best practice would be to go to Splunk documentation size go to Splunk enterprise administer and admin manual because I’ve already mentioned in my previous tutorials these manuals, I always keep them handy.

So that because I can get all this configuration references from here. The indexes configuration is always stored under indexes. com all the configuration related to adding an index, deleting an index, changing the size of the index, changing the location of your index index file all this configuration is present in indexes conf. To add a new index using configuration files, we need to add a new file. If there is no indexes conf, it’s already present under local directory. We will add a new file under system local just for example of understanding the indexer but in the future we will see how we will be placing or creating these indexes using our deployment. So to move on, I’ll click on indexes. com example there is one database that has been created hatch I’ll copy the hatch one, I will not modify anything. If you want we can modify the name but since we are just demonstrating I’ll just copy including the name localindexes. com whatever we have copied from the configuration file reference I just copied it. We have not done anything. We have just copied and sample index which was created using this and if we want we can rename the name of the indexes but we are not going to do deal much with this indexer so let us keep it as it is. These locations HomePath Cold Path and Tower path will come to this later probably with a deeper understanding about Splunk and indexer and the clustering part and the retention policies where this play a key role.

 As of now, for simplicity of configuring index, we’ll stick to creating index for now, this is the regular syntax. We have created an index using configuration edit of indexes conf. Let us restart our Splunk instance shouldn’t take much time. Yes it is starting. This should be our indexes. Yes, this is our indexer login. So we are under indexes correct. So where is our new hatch DB? Yes here it is hatch DB which we have created using yes configuration edit we have created Windows using web linux using CLI and Hatch using configuration edit we have created all this indexes but where do the actual location go? But we all know by now the DB location of Splunk is opt splunk where lib Splunk this is your default database location on Splunk where all the files that have been received, processed and stored here under this you will find lot of directories out of which your even newly created indexes are present. Let us go through our windows index which we have mentioned one is hot DB, cold DB and Tau DB. We will understand why data model summary is present and what is cold DB and what is Tau DB when we move to retention rolling of buckets and what is a data model and how does data model summary works? For simplicity of configuring an indexer, this will be your index location and all the files will be stored here at any time. If you want to check your size of the indexer, you can check the size of the folder or can just go to your indexer GUI and check for those sizes.