AZ-303 Microsoft Azure Architect Technologies – Monitor Azure Infrastructure Part 3

7. Log Analytics

Azure captures an incredible amount of log data and as such it can be difficult to find and visualize what you need. For this reason, you can create a Log Analytics Workspace, which is a powerful tool for querying those logs to find the information you need. The Log Analytics Workspace is also required to provide other management capabilities such as advanced report and update management. Log analytics can query across many different data sources, including Windows event logs, Windows performance counters, Linux performance counters, custom fields, custom logs and so on.

You can even configure custom event logs that can be queried by your Log Analytics Workspace. Log analytics uses its own query language. So when you build a query, you start by determining which tables have the data that you’re looking for. Each data source and solution stores its data in dedicated tables. In the Log Analytics Workspace, documentation for each data source and solution includes the name and the data type that it creates and a description of each of those properties. Many queries will only require data from a single table, but others may use a variety of options to include data from multiple tables.

The main tables often used are event, syslog, heartbeats, and alerts. The basic structure of a query is a source table followed by a series of operators separated by a pipe character. You can chain together multiple operators to refine the data and perform advanced functions. So for example, this query returns a top count of the top ten errors in the event log during the last 24 hours. The results are then displayed in descending order.

Other common operators are Account, which returns the number of records within the record sets limit, which returns up to the specified number of rows a summarize, which produces a table that aggregates the content of the input table. So this is great for doing counts of records, average prices and supplier information and so on. And Top returns the top N records sought by the specified columns. Because of its flexibility and power, it can take quite a lot of effort to invest and set up all the queries you might want to support and monitor. Log analytics provide an entire series of solutions. These solutions are prebuilt sets of queries and dashboards for common scenarios.

8. Log Analytics Walkthrough

In this lecture, we’re going to create a Log Analytics Workspace, connect it to a virtual machine and then start queries from the data in it. The first thing we need to do is go ahead and create it. Perform the search for Log analytics workspace and click Create. As usual, put this in a resource group and give it a name. Next we choose the pricing tier. Just go for the default which is a page you go. Basically with Log Analytics you pay mainly for ingestion and storage and again give it any tags and click Create. Once that’s completed, rather than go to the resource itself, I’m going to go to my Virtual Machines Blade and I’m going to select my Windows Virtual Machine. I’m going to go to Logs under monitoring and then I’m going to enable Log Analytics that will either ask me to choose or create a new Log Analytics. So the default is it tries to create a new one, but I’m going to go and use the one we’ve just created and then I’m going to go and click Enable. What this will do is now go and install an agent on the virtual Machine and connect it to the Log Analytics Workspace and that will take just a few minutes. Okay.

Once that’s installed, we then need to connect to various different kinds of logs that we can send to our Log Analytics Workspace. The first will be in the Virtual Machine pane. Again. Go back up to Activity log. And whereas we can use this pane to look at operations that have acted on the Azure Virtual Machine, we can also tell it to output these logs to a workspace. So if we go to Logs and click Add, we can select the workspace we want to add it to and that will then start outputting the logs accordingly. So that means these logs that would normally appear here would then go to there. If we go to Diagnostic settings, we can also emit diagnostic event information. If we click add diagnostic settings. Again, we can set a number of logs that would be output, give it a name and tell it to send it to Log Analytics and again, choose the space that you want and then click Save. So the point with this is although you can access all this information directly in here, by sending them to a Log Analytics workspace, we get a single place where we can monitor these things. It’s also important to understand that all this information that it’s emitting is from the Azure side of the analytics and performance metrics. So when we installed the agent on the virtual machine, that then gave us the ability to get information directly out from within the host itself. But again, we then need to go and start to configure that side.

So once all that’s done, go through and go to Resource groups, find your Log Analytics resource Group and open the Analytics Workspace you’ll see two things called solutions here and we’ll come back to those shortly from here. What we want to do is go down the left hand side and if we go to virtual machines, we can see the virtual machine listed there saying that it’s been analyzed by this Workspace.

So if you have multiple virtual machines, this is another way in which you can add the Log analytics to them. What I want to do now though, is go to Settings and Advanced Settings and through here is where we can start to define what information we get out of the virtual machine. We can also see these agents. So if you have virtual machines on premise or even in another cloud service, you can download this agent, install it onto them and then give it this information of the Workspace ID and the primary or secondary key. And therefore you can start to capture information from virtual machines that are anywhere. See what that actually looks like if we log onto the virtual machine. If we go to Control Panel, make sure the view is on small icons, and we now have an extra option in here called Microsoft Monitoring Agent. So this is the agent that got installed when we enabled the Log Analytics workspace. If we open that up and along the top we say Azure Log Analytics. We can see that that’s automatically connected this up for you.

So if you wanted to do this with an on premise virtual machine, all you would do is download that agent, install it manually, and then in here click Add and give it the Workspace ID and Workspace key. So the next thing we want to do is actually tell it what information we want to start collecting. So down the left hand side, if we click Data and with different things that we can capture. So for example, we can capture is logs. For Linux machines you can get Performance counters, and for Windows we can get event logs and performance counters, but it doesn’t add them by default. So you have to tell them what to do. So let’s say on our virtual machines we want to collect all application logs. We’d go to the event logs here, start typing Application and then we can either add individual application logs or just a general application log. Click add and then click save.

Again we can go to the Windows performance counters and again we can add individual performance counters, which again is information that gets emitted from within the actual host itself. So we can either add our own up here or we can add the suggested ones and again click Save. So the point around all this is that allows you to add just the events and performance counters that you need for the workload that’s running on that virtual machine. So for example, if you had a SQL Server running on it, you might want to add forms counters that are specific to SQL. It’s also possible to capture information from custom logs and fields. So to get an example of this, let’s go back into our virtual machine. And if we go to the C drive windows Azure and Logs. So here we have some logs that get emitted by when the agent was installed and when the virtual machine itself was built.

So we’re going to get an example log out here, which is this Agent Runtime log. So what I need to do first is I need to get an example of the data in that log. So I’m going to open it up, just get a top few rows and copy them. Now, back on my laptop, I’m going to run Notepad and I’m going to paste that in and I’m going to save that file and I’m going to call it Agent Runtime. And I’m going to go to my custom logs. I’m going to add a custom log. So the first thing I do is I give it that sample file I’ve just created that then opens it up to and analyze it to see what kind of information we get. And then we tell it where the actual log path is on our virtual machine. So if we look in here back to our virtual machine, we can see it’s in here. So just copy that path and then the actual file is Agent Runtime. It’s called Agent runtime. TXT. Click add and then click next. Give it a name and click Done. So what will happen now is that this Agent Runtime log that’s within our virtual machine will get read by Azure and ingested into the log analytics so that we can start querying the data across it along with all the other information. It can take an hour or so for it to start ingesting information. So give it 30 minutes to an hour and then come back and continue the lecture.

9. Querying Log Analytics Walkthrough

Virtual machine has been running a while and we start collecting some logs we can go ahead and start analyzing them go back to our logs and go to the logs blade so within here we can start querying out our logs. Now, down the left here we can see there’s some of the logs we can look at. So we’ve got the log management, we’ve got the VM insights logs that we can query and in fact, we’ve got the custom logs. Within there we can see our agent run time that we added earlier. But let’s go for something simple. First of all, when we write now, queries, the first thing we do is we tell it what? Table we want to query, so I’m going to query the event table. Then we use a type because I’ve pressed return it’s off to put the pipe in, but I’m going to say where. I’ll choose a field, so computer equals and then we’ll type in the name of our computer. And then we’ll go ahead and click run. So that then returns all our data. We can see there’s quite a lot of information here.

So one of the things you might want to do now is filter that. Let’s filter it on event level. As you can see, we’ve got quite a few different event levels, with four being one of the higher ones, so we could then say another pipe. And again, say where. This time we can say event level is greater than or equal to four. And run that again. And then it’s filtered all our events now. So we’re just seeing events level. Four. Finally, what we might also want to do is summarize those results. So although that might be useful, what we might want to see is to see where these events are coming from, so we can then say summarize.

The count and then we can say by source and then run that now we get to see where the actual errors are coming from and how many errors they are. Again, although that’s very useful. It’s not very friendly. So the next thing we can do is turn it into a chart. It’s defaulted to a stack. So maybe we can say, well, actually, we can change that chart into a pie chart. Then we get a much better visualization for our report. What we can now do is go and save that by clicking the save button at the top just there. Let’s give it a name and we’ll give it a category save that now appears if we click at the top here in this Query Explorer and it appears in the saved items there.

So now we can quickly and easily get back to those event sources as we need. What can also do with this? Is actually pin it to a dashboard. So again, if we go and say pin to dashboard, we can then go and create get the dashboard we found earlier and pin it to that dashboard. If we click example queries up here, we can actually see a history of our queries, but we can also see a number of examples. So for example, if you want to see the memory and CPU usage across all your virtual machines, go to this performance and click Run and it brings up an example query. And again it shows us all this information. And again we can adapt to a dashboard or even modify the query to be what we want it to be. What I’m going to do now, I’m just going to go back to my original query and I’m just going to copy that. Because when we’ve got all these different elements, sometimes it’s handy to have an overall report that brings a number of different queries together.

So for example, if you want to see lots of information about your computer, so events and general performance, you might want all these groups. So what we can do, we can group them all into what’s called a workbook. Down the side here we’ve got workbooks. And depending on what we’re looking at, with a number of built in workbooks, we can either choose to edit or we can start with an empty one. I’m going to go and look at the agent health workbook. This agent health workbook is quite a handy one because it shows you how many nodes we’ve got our agent installed on and whether that agent is healthy or not. But I might want to add our event source information to that one.

So what we can go and do is say edit and if we scroll down to the bottom, we can add a new one here. So let’s say add a new item, we’ll add a query. Let’s paste in the query that we did in our workbook earlier. Let’s just run it to make sure it’s working okay.

Again, we get all the information we want to change now how it visualizes it. So we’ll change the visualization and we’ll say pie chart and then we’ll click done editing. Click done editing at the top. So this is a great way to build up your own workbooks that give you dashboard of whatever your team needs to analyze as part of their day to day working. Log analytics are extremely powerful and the important thing is to understand how we hook up various components to them. In the next lecture, we’ll look at another type of logging, which is app insights.