SPLK-1003 Splunk Enterprise Certified Admin – Splunk Post Installation Activities : Knowledge Objects Part 5

21. Creating Macros in Splunk

The next knowledge object in our discussion is the macros. The macros in Splunk are similar to macros in Excel where it is a small piece of code which will be reused multiple times. Instead of rewriting all the search query here we will be using macros to reuse the search of multiple times in multiple places. In this example we will see how to create a small piece of macro, how we can share these objects to other Splunk users so that they can start using a simple macro instead of creating a separate and longer searches. This is a searcher. Let me write a short query that is index is equal to means source type is equal to our access log. Source type is equal to access log status is equal to 200. That is successful results and I will get location of our IP address that we have extracted using props and I’ll get top countries using those IP address. So let us hit enter. I’m getting top ten countries for my website that has been accessed and which were successful results. I got top ten countries.

Now every time I need these results, I don’t need to rewrite this search query or any place wherever it is required. So what I’ll do, I’ll just copy this search and I’ll go to settings advanced search. In this place you will also be able to see if any macros which are already being present that you’ll be able to use. By default, there will be few macros that are just use it for searching in Splunk. So here we have two menu search macros and Search commands. Search commands will come at a later part. We’ll go to search macros. In this we’ll be able to see some of the internal macros which are created by Splunk for its internal searching purposes.

These are some of the searches with Splunk internal searching users regularly. So it has made it as a macro and it has stored it so even as a user you’ll be able to use it. We’ll create our own macro, that is our top countries that we got from this long query. What I’ll do is I’ll mention this as top countries and I’ll paste my search query. If I have any arguments, I can mention it here so that the arguments can be passed as part of your macro. As of now, we don’t have any arguments for our simple search query. So I’ll place it as it is. I’ll click on Save and once it is saved, as you can see it is private by default. If I want to share with other Splunk users, I can go to share whether I can share within this app or I’ll share with globally. That is everybody accessing splunk Everybody has read permission admin as Edit permission and also Power user as the edit permission.

I’ll click on Save and once it is saved, go back to your search query. Instead of writing the complete command. You can call this by using macro that we just created. To call the macro there is symbol that is the same key as your tilled symbol but you need to push it with escape. So it is just below your escape character which holds your tail symbol but it is usually called as a backtick. Let me see if I can find this in my on screen keyboard. No, not this. So it is basically Till symbol. But if you are entering till symbol you have to press down shift but instead without shift it will be present on the same key. It is known as the backtick. Let me see if I can show it to you in notepad. So this is the symbol. Basically this is your back tick. This is known as backtick which is as part of your tilt key. Enter this and enter the macro name that we have just created and close your back TAKE.

So once you have entered this, your search will be able to load instead of complete query. See? As you can see we are just called our macro function. We have defined it here top countries. So we called it using this back take suggesting Splunk that this is not a search query but this is a macro. So Splunk invoked this macro in turn to give us the actual results. What we were looking for. Once we have created macros using GUI, you should also have knowledge where these files are located. Let us go to our searcher. So this is our searcher. I am in the local directory.

As you can see there is a file created macros. com which holds the information about the name of the macro and the search used by the macro. If you have additional arguments and other values it will be on the next line, same argument and it continues with additional eval command validation. And further, this is a simple macro where it shows definition and the search of the macro. To summarize a couple of knowledge objects event types that are stored in event types conf, tags conf and macros conf. All names are relatively simple. It sticks to the concept that whatever the knowledge object we call them the same configuration file. It holds this information so that anytime if you see these modifications and if you want to see what it has stored sometimes a special character or and symbol might be differing well story. So make sure everything where to look for this information as part of CLI.

22. Searching in Splunk

In this module we will be understanding on how to search in Splunk. We’ll be seeing some of the basic operation as part of everyday searching in Splunk. Now we have our on boarded data, our Splunk installation set up and we have created some basic knowledge objects. Now we will see how to search efficiently using Splunk. This is our searcher. We know what our index is. So I’ll go to index main and I’ll just hit enter for the last 30 days. We have relatively small data, so it should load up pretty quick. We have this much information as part of our Splunk for the last 30 days. There are lots of information on our fields. I would highly recommend going to our first chapter where we have discussed the complete UI of Splunk and described each and every fields that are available.

As per this part of the tutorial, we’ll be completely following up upon that and we’ll be strictly focusing on how to search in Splunk. Now I have a narrowed down the index. If I want to search for Windows, I will go for index is equal to Windows. This is based on the during installation. Whichever the lock sources you sent to that particular index, you’ll be able to search the same. We have index is equal to main we are sending that is the default index, all the logs. So we’ll be sticking to index is equal to mean even though you can also search using Wildcard, that is star. You can search the star and you’ll end up getting the same number of results. Because it will be searching our default index itself. Let us go. And this is the best way to narrow down the results. Instead of plain old Wildcard, I’ll first narrow down my search to index is equal to main in index is equal to main. I have two host. I have something called as, my universal forwarder laptop and Arun Kumar PC.

So I’ll click on this host, which is of my interest, and I’ll see what all the information it has collected. Let’s say I need to see for the error. What do I type? I simply type free from error keyword. So what happens when I type error? This search query, it is looking under main index for a host named this and with a string containing errors. All the logs with this error with this error keyword are matched and displayed in front of you. What is happening here is we have not mentioned anything, but Splunk takes it as implicit and so that if you don’t mention anything, that means you are searching for both condition. This is how Splunk interprets if there is nothing in between different phrases it considers you are looking for both these values in your logs.

That is how our search query works or interpreted by Splunk. So there is another function or with this logical function I’ll search for error or warning message in this log. This is an and condition. Within this it is looking for error or warning scenario and it will display all the events that matches the specific criteria. And also here you have other operations like not which shows only error, not warning. Let’s say this should make more sense. I’ll search all the logs in my previous criteria, but not warning messages or not info, which again narrows down to only my error flocks.

As you can see here there won’t be any values that contains warning or info. By now we know that whatever the freeform search you do either caps or small, that doesn’t matter. But it only matters what is the phrase you are typing, whether it’s with wild card or not. But if it is a field name, it should be always whatever displayed here the field is case sensitive, but the value is not case sensitive. Now we understood and or not, let’s see how other fields work.

23. Search Modes in Splunk

Let me go back to my main search and this time I’ll go for my access logs. So this is our access log, which is our web server, which has many number of fields. We have two more conditions which we can verify how to specify greater than or equal to values. Let’s say I have status fields in my logs, so let me make it selected so that it pops up right here in our screen of viewing status field. It has 200, 504, hundred and many different values. Let’s say I want status equals 200. It displays only 200 values, nothing special about that. But if I choose status greater than 200, it ignores all the 200 values and displaces rest of the values. As you can see 50348 and 500 which all represents which are status values greater than 500. And let’s say I need to filter out my 505 and five, three and 500. I’ll use the implicit and since we didn’t mention anything here, it will consider it as and so I’ll mention status should be less than 500 or you can mention and condition specifically so that Splunk interprets as we want by default. I’ll leave it as it is. It treats it as and condition.

And we have five values now, which is completely 400 values. We have learned about greater than and less than. Let’s see how we can specify status is equal to or not equal to 200. This is how we specify not equal to condition exclamation followed by equals and followed by the value which should not be matched. Let’s hit enter now go back to status field and we can check there is no 200 value. Let us eliminate 500 with some wild card. I’ll use status exclamation equal to phi star. So what it does is anything status field that starts from phi will be eliminated. Out of the result we’ll go back to our status field again. As you can see, we have only our 400 related errors. This way you can narrow down your results much much further. The fastest way would be to click on some field, click on the value, it automatically updates your search.

In case if you want not match condition, you can click on or update your search to match it as not. Also this is one way to add a not condition. Also you can add boolean not which is the same function saying that don’t match wherever action is equal to add to cart. These are some of the methods where you can fine tune your search and populate the results. And we have also seen what are fast mode, smart mode and Verbus mode. Just remember, fast mode is the fastest of all. Let’s modify the same search and we’ll see what all the changes will reflect. So, fast mode. I’m selecting fast mode here you’ll be able to see hardly ten fields. It has completed the search, but we have very limited information in our fields. Menu.

Even if you extend you have only few fields but there is a lot more information in the locks. The Fast mode is more focused on getting you the results faster rather than parsing it completely or giving you the complete information of the fields. The Smart Mode whereas gives you the complete information of what is requested. So in this search we have requested only for events. We have not specified any visualization. Let’s go ahead and mention some visualization saying Top action values. I’ll mention top is equal to action? Top is a command.

Action is a field name. It says the most value for the action is the view component. So now we have got visualization from our Smart Mode which automatically produces a visualization. We need to make sure the recommended charts are set to see the value. Now we have a Smart Mode which gives automatically the visualization and the Statistics as requested by our Top command. But it will not give you any events. But Smart Mode gives you whatever it has been requested. So from using Top command we are requested for a Statistics and Visualization which is populated by default. So if you want event to be seen, it clearly says you need to search in Verbose Mode. Verbose Mode is like every process utilization and it gives you everything. It gives you along with events, all the fields extracted along with Statistics and Visualization. So this is just a quick overview but if you’d like to know more about this, I would highly recommend you go back to our first module of the lecture where we have gone through all these search modes.

Now we have known about search modes, let us see how we can validate the data that has been uploaded to Splunk is being parsed properly or not? So the best way to check for parsing of the data is let’s say this is our logs. Make sure you are running in Smart Mode. This is our uploaded log. We know this is a complete line as a log. Once you have uploaded, you can see from notepad or any other you’ll be able to notice the difference between the lines. This is our actual log. And if you expand this arrow, you’ll be able to see all the fields extracted by this log. In case any of the fields have been misnamed or wrongly named, you’ll be able to notice and we can consider them as not being parsed properly. And also if you see any of the values that has not been extracted. Let’s say I’m able to see the value 159 here but I couldn’t find anywhere here. That means the logs have not been passed properly or completely. We can consider this as a response time and we can create a new field so that our logs are passed completely. This is a quick to identify whether logs have been parsed completely or not.

24. Creating Alerts in Splunk

In this module of our discussion we will be seeing how to create alerts in Splunk. In our previous modules we have learnt how to do basic searching in Splunk and how to use couple of boolean operations and or not to filter out the necessary events that we are looking for. Some of the topics our concepts we are going to study about alerts are first one is creating a search query that is the important part of your alert creation for what condition you need. This alert to be created will be defined as part of a search query and also when this alert should be run, whether it should be real time or a scheduled alert and what should be the action taken by this alert and should we enable throttling and who has the visibility of these alerts and who can modify these alerts? If the action is emailing, should we email the link or just the results are as an attachment?

We will see all these operations in our exercise. This is our search ed. We can start with searched. So let us consider an example. Whenever my server throws either 400 or 500 errors as per our Http code, we know that something is wrong with our server or some page has not been available for the user. Let us see I’ll search for my index main where my data is present and the source type of my web server logs are access. I didn’t mention the complete access combined with cookie. That is our source type and status should start with either 500 or status should be starting with 400 something. This typically means there is something going wrong with my server or the client requests have not been successfully processed. Let us validate.

Here is our status field where it shows 503. That is our internal server error. Similarly 408 and other server errors which are defined as per our Http status code. In fact, in our previous lecture we have also learned how to use our Lookup field to enhance the data. Let us use the same look of field so that we will be able to make more sense out of the data output description of the status field. This is not input. This will be your Lookup command. There is the description yes, here it is. Let me select it here. As you can see, these are the error description as per our status.

Let us see how we can create an alert so that whenever this event occurs we should be alerted almost at the real time. I will write a query where to display just the statistics based on the count of those statistics status, values so stats command I’m using it to just display this should be count by what it does basically is it sorts everything based on your status and how much count of events that has been received in a period that you have selected.

Now we have set up our query. This is our condition. We should be alerted whenever these status quotes are received on our web server. Click on Save Alert and here you will give a short title where it displays a brief information of what this alert does. And here you can probably write web server generating 405 hundred errors and the permission when you see permission it is either private or shared in app. If it is private, the alert will be only visible to you and if it is shared in app, anybody using this app will be able to access.

25. Splunk Alert Condition and Sharing

So let me keep it shared for now. So the next option is alert type how it should be specified, whether it should be scheduled to run at every specific hour or day or week or month. So since it is a server performance, we’ll keep it real time and also the trigger condition. What should be the a trigger condition? It should be each result should trigger an event or is it based on custom? We can also keep it something like count greater than ten. If this count is increased or more than ten these errors in our logs, then it will trigger an alert. Similarly we’ll keep it like if I have ten host web servers, all are throwing me errors you can keep if the number of hosts in my criteria is greater than phi, that means something is going wrong with my web servers. All the servers have started to throw errors. These are some of the trigger conditions you can specify as of now since we have sorted it based on count, whatever the results become will be triggering. The next option will be the throttle. In throttle you can suppress the event for specific amount of time.

Let’s say I’ve got an alert, I’ve got some people who are looking into this issue. So I don’t need this alert to be triggering containing the same status value. So this is my field. Even I get one value that is 503 alert got fired and somebody is looking into it. And again if the status field throws 503, it will be suppressed for 60 minutes. So that we give them 1 hour window to fix this issue or identify what is the cause of the issue and take remediate actions. This is how the throttling works. Once you have set up all these things, there is an add action field. The add action you have multiple actions where you can add this to triggered alerts so that you’ll be able to see at a later stage how many alerts have been triggered and this will be stored under your Splunk activity menu. Similarly, you can log this event so that it can send a log to another Splunk instance where it can receive and process this triggered alert. And similarly there is an option to run a script.

Let’s say whenever there is a multiple 503 I will restart my server. So you can probably write a small python or a bascript and invoke this using your run script and mention the script name. So this is one of the auction. The next action will be send email which should be very basic, that is if you want to add the sender’s list, you can add it separated by a comma and if you want to add CC and BCC you can mention those details. Set a priority for your email. Give a short name which represents the subject of your email so that the action can be taken and complete action plan or the message why this alert was triggered and the alert can include a link to your results, that is if they have access to Splunk. If they don’t have access to Splunk, you can uncheck this and you can send them a CSV file containing the results or an inline table which displays the results.

Whatever you are seeing in the background, the statistics as part of your email and you can also include your search string trigger condition and what time the event was triggered. So all this you can choose whichever are necessary and you can uncheck whichever are not necessary. So the final option will be type. So in this option you can choose either plain text if your emails are not supported, HTML format if it supports go for HTML and plain text for better formatted output of the alert. So here I’ll just add my email so that if any alert triggers I’ll be getting an email notification and I’ll keep everything as default. I’ll save this alert so once it is set you will get a permissions tab.

If this is a search schedule search, it says since we are running enterprise trial license it will not be run after the license express. So if it is not then if we have an actual Splunk license this warning will not be popped up. The next is the permission step in which you will be able to give privileges to the people who can edit or read this alert. As you can see Admin and power as the edit privileges and all other people using Splunk as read permission. I’ll just to make it global, I’ll make sure this alert is visible through all the apps.