SC-200 Microsoft Security Operations Analyst – Threat Hunting in Microsoft Sentinel

1. Threat Hunting Concepts in Microsoft Sentinel

Hello everyone and welcome back to my course microsoft Security Operations Analyst SC 200. We are starting the last section of the course and in this one we are going to discuss mainly about threat hunting in Microsoft Sentinel. Now we are going to first of all, in the first lesson we are going to establish some threat hunting concepts in Microsoft Sentinel. We are going then to talk about threat hunting with Microsoft Sentinel and we’ll provide some examples. And lastly, we are talking about notebooks in Microsoft Sentinel. So let’s get into the first lesson here and discuss about some threat hunting concepts. Now, the term threat hunting is defined differently by different people. This is from the start. The most commonly used definition is the idea that you are proactively hunting through your environment for a threat or a set of activities that you have not previously detected.

The so quoted not previously detected part is what differentiates threat hunting from incident response or other triage or threat detection. Let’s see now other users use other uses of long term of the term threat hunting include searching for threats with newly obtained indicators. So if a threat intelligence feed provides a new IP address considered harmful, an analyst can take that IP address and search through the logs to find if the new indicator was seen in the past. So technically this is not threat hunting because you are using a known bed such as an IP address to look in past events. Microsoft Sentinel already provides hunting queries to facilitate this process so you can hunt for more evidence based threats from a current incident or alert.

As part of the incident analysis process, it is vital to explore data based on evidence found in the current incident. Both Sentinel and Microsoft 365 Defender provide this type of hunting capabilities. Now, all of these approaches have one thing in common they use KQL queries to find threats. Microsoft Defender and Microsoft Defender for endpoint are more focused on indicator and analysis types of hunting. Right, like we’ve established in the section where we discussed about Microsoft 365 Defender and Microsoft 365 for Endpoint. But Microsoft Sentinel rather provides more features to manage the threat hunting process. First of all, threat hunting should be a proactive process and as you can see here on the slide, I’ve basically put down like a workflow for threat hunting.

So why would you do proactive hunting as you hunt for not previously detected threats? The concern is that if you wait for the threat to be detected, the compromise impact could be more significant. So if we don’t have a known indicator, then what are we hunting for? We hunt based on hypothesis. So the hypothesis might start with operational threat intelligence, for example, and then list the attacker tactics and techniques. A hypothesis can search for a specific technique, not an indicator like an IP address. For example, if malicious activity is identified, we might have discovered the attacker earlier in the attack process before they have an opportunity to exfiltate data. For example. Now, threat hunting should be a continual process. So we start at the top of our cycle over here.

So we develop our hypothesis and our hypothesis helps us plan out what we are going to hunt for, which basically queries require us to understand where we are going to hunt and how will we do it. This means that we need to understand the data that we have, the tools that we have, the expertise that we have, and how to work with them. Now, the hunting cycle doesn’t stop when we execute the hunt. There are still several phases we need to conduct throughout the life cycle, including responding to anomalies. Even if we don’t find an active threat, there will be activities to perform. Now, routine activities in the hunt cycle should include setting up new monitoring, improving our detection capabilities. Now, everything done in threat hunting should be documented.

Documentation for threat hunting should also include the what, the how and the why, input and output data, how to replicate the hunt, and next steps to be taken after the hunt is actually ran. Now, hunting starts with hypothesis. As I’ve mentioned, the idea of what we are going to hunt, this is the hypothesis. Getting this right is critical because it drives our focuses on what we are going to do. What makes a good hypothesis? Well, there are many factors, but here are some key ones. Let’s see. So first of all, keep it achievable the first success factor for a good hypothesis. So don’t perform a hunt where you have no hope of finding results because you do not have the data available or having sufficient knowledge about the threat to understand how to find it.

Then you should keep a narrow scope, avoid a broad hypothesis such as I am going to hunt for strange logons, right? Such hypothesis will fail to define what results could mean. Then keep it time bound. Here the questions are are you looking for any login since the beginning of your logs? Are you looking for last week or for the last day? The time bound process also is used in documentation. You will want threat hunting to be a continual process. If you don’t time bound your hunts, there is a chance that you will end up just repeating the same hunt on the same data sheet. So you’ll be able to say I did this hunt at this time, covering this period. With this documented, your team members will know what period was hunted for the specific hypothesis. Keep it useful and efficient.

You want to target threats that maybe you don’t have adequate coverage for in your detections. This might be things that you know that you’ve previously missed or that you haven’t detected. A good Security Operations center team typically has a good idea about where their coverage is good and where it may be weaker and needs improvement. You also want to make sure it relates to realistic threats. There is no point, for example, in hunting for an advanced threat that targets industries that you are not in or a platform that you are not using. And then the last key point here is keep it related to threat model that you are defining against. Otherwise you may spend much time threat hunting for things that you will never find and which are not a threat.

So don’t start your threat hunting journey going after the most advanced threats. Start with the basic and incrementally mature your organization’s threat hunting capabilities. Start with simple hunt hypothesis. An example hypothesis could be that we have a threat intelligence that a threat actor has automated attacks that use the CMD exe process, right? And another good hypothesis would be we want to check for the last day in which accounts have run CMD exe but have not run CMD exe during the past week. So these are just a few examples. You will also will have links to further documentation in regards to this concept in the downloadable resources for this lesson.

We are wrapping up our lesson with this and I’m going to see everyone in the next one where we’ll discuss about actually doing threat hunting with Microsoft Sentinel. Until then, I hope this has been informative for you and thank you for viewing.

2. Threat Hunting with Microsoft Sentinel

And welcome back to my course. Microsoft Security operations analyst SC 200. Now, in this lesson we are going to discuss about actual threat hunting in Microsoft Sentinel. Microsoft Sentinel contains powerful query tools that can help you as part of the Security Operations Team, security Operations Center team to find and isolate the security threats and unwanted activity in your environment. You can basically use the search and query tools in Microsoft Sentinel to hunt for security threats and tactics throughout your environment. Hunting Queries basically enable you to filter through large amounts of events and security data sources to identify the potential threats, to track down known or unexpected threats. Right now, the hunting page in Microsoft Sentinel.

And this is how the hunting page looks. I’m going to show it to you in the portal as well has built in Queries. Now these queries can guide your hunting process to help you pursue the appropriate hunting paths to uncover issues in your environment. Hunting Queries can expose issues that aren’t significant enough on their own to generate an alert, but have happened often enough over time to worth some investigation let’s say. Now the Hunting page also provides a list of all the Hunting queries, the built in Hunting queries that are available. You can filter and sort queries by name, provider or data source results or tactics. Now you can save these queries by selecting the favorites icon, right. And they will go also to your BOOKMARKS over here.

And when a query is selected as a favorite, it runs automatically each time you open the Hunting page. That’s one thing to note well how you manage your Hunting queries. When you select a Hunting query, the query details will appear on a new pane. So for example, let me just quickly hop into the portal and I am on the Hunting page over here. And as you can see this is the list of the built in Hunting Queries. From here you can filter and you can filter by all of these nice filters provided over here. And again what I wanted to show you when you select a Hunting query. So for example, rare audit activity initiated by user in the right hand side here a detailed Spain containing the description and the actual code over here will open and of course you’ll find other information.

Information include related entities and identify tactics. So you can run the query interactively by selecting the Run query button here and I am going to run it now and you will see that this will change from three queries ran to one query. So this is the way you run the query interactively. You also have available the Run All Queries button which will run all of the 162 queries available and you can view the results after that by clicking on View Results. For example, if I click on View Results this will actually open up in Log analytics over here. And of course you can see the results over here, right? You can save your query from the Log analytics page.

And again, if I just go back here and I select the Rare Activity query that I’ve just selected over here on the right hand side, if you click on these three dots, you can add it to Favorites, you can clone the query, or you can add it to a live stream or directly create a detection analytics rule from this particular query. Now, if you want to create your custom queries so let’s say you are not interested in the built in ones, although you should, well, that’s very simple. You click on the new query button over here and this will take you to this page where you give your query a name, a description, you specify your actual KQL query that you want to run and of course you can map entities over here and select the tactic for your custom Hunting query.

Now again, you also have available the queries repo on GitHub and you will have a link for that in the downloadable resources for this particular lesson. Now again, to hunt for threats in the environment, you will have to review large amounts of log data for evidence of malicious behavior, of course, and let me just quickly go back to my slides over here. And of course you can use BOOKMARKS to save your queries, let’s say. And this can help you hunt for threats by preserving the queries you’ve ran in Microsoft Sentinel along with the query results that you deem relevant. You can also record your contextual observations and reference your findings by adding notes or text.

The Bookmark data is visible over here, under my BOOKMARKS, over here. Of course, we have no BOOKMARKS available yet because we haven’t created one in this trial environment. But this is where you’ll find the BOOKMARKS. Now again, directly from the query, you can run it, you can add it to Favorites like this, here we go, and it will load up over here, right? And you can also create an analytics rules, as I’ve mentioned, to have an incident assigned to your query when you find results. Now, you can use the investigation graph, for example, from within an incident to explore your BOOKMARKS in the same way you would investigate incidents in Marksoft Sentinel.

So from the Hunting page you can select any kind of Hunting query and you can perform an investigation to open up an investigation graph and basically get to the same visualization experience that you saw when investigating the incidents. Now, let’s talk about observing threats over time with Livestream because we have yet another service over here. Let me get back to the slide. You can use the Hunting live stream to basically test your queries against live events as they occur. Livestream provides an interactive session that can notify you when Microsoft Sentinel finds matching events for your queries. A live stream is always based on a query typically you use the query to narrow down streaming log events, right? So only the events that are related to your threat hunting efforts will appear.

You can use the Live Stream to test new queries, to generate notifications for threats, or to launch investigations. Livestream refreshes every 30 seconds and generates an Asia notification of any result from the query. To create a Live Stream from the Hunting page, you select the Live Stream tab as I’ve shown you previously, and you click on New Livestream from the toolbar. The Live Stream queries run continuously against your live environment, so you can’t actually use time parameters in a Live Stream query. Right on the Live Stream page, you specify the name of the Live Stream session. The query that you want to result for and notifications for Live Stream events will appear in your Azure Portal notification.

You can play Live Stream to review results or save livestream for later references. Saved Livestream sessions can be viewed from the Live Stream tab on the Hunting page. And you can also elevate events from a Live Stream session to an alert by selecting the events and then selecting to Elevate to an alert status. And now let me actually show you how to do that. So if we take this query, for example, and let me just copy it from over here, and if we go to the Live Stream tab, we click on New Live Stream, let’s say test Livestream as the name. We specify our query over here and we select to play the Live Stream. Okay, we have a time range here in our list, so let me actually select a different query because again, as I mentioned, you cannot specify time range parameters in the Live Stream as it runs continuously.

So let’s say this one, this will look for any kind of deletions in the Azure activity log. As you can see, you can create an analytics rules from here and you can play the Live Stream. And this is running and you will get notifications whenever it finds a match. Of course it won’t find a match because we don’t have enough data in our environment for us to actually get a result. So I’ll pause this now. This concludes our discussion for this lesson. I am going to see everyone in the next lesson and the last lesson of the course where we’ll discuss about hunting for threats using the notebooks capabilities in Microsoft Sentinel. But until then, I hope this has been informative for you and thank you for viewing.

3. Notebooks in Microsoft Sentinel

Everyone and welcome back to my course microsoft Security Operations Analyst SC 200. Now, in this last lesson of the section and the course, we will discuss about another concept in Microsoft Sentinel called notebooks. So in Microsoft Sentinel, you can use yet another tool or capability, however you want to call it, called notebooks. To perform threat hunting, a Jupiter notebook allows you to basically create and share documents that contain live code, equations, visualizations or explanatory text. Uses include data cleaning and transformation, numerical simulation, statistical modeling, machine learning, and much, much more. Jupiter extends the scope of what you can do with Microsoft Sentinel data.

It combines full programmability with vast library collection for machine learning, visualizations and data analysis. Now, all of these attributes basically make Jupiter a useful tool for security investigation and haunting. Several notebooks developed by some of Microsoft security analysts are already packaged within Microsoft Sentinel. Some of these notebooks are built for specific scenario and can be used as is. Others are samples intended to illustrate techniques and features that you can copy or adapt for your use in your own notebooks. Other notebooks may also be imported from the Microsoft Sentinel’s GitHub community. Now, notebooks have two major components the browser based interface, where you basically enter and run queries and code, and where all the execution results are displayed.

And the kernel, the back end that is responsible for parsing and executing the code itself. The Microsoft Sentinels Notebooks kernel runs on an Asia virtual machine. Several licensing options exist to use more powerful virtual machines if your notebooks include more complex machine learning models. For example, the marks of Sentinel notebooks use many popular Python libraries such as Pandas, Map, Plot, Lib, Bokeh and many, many more. There are great many other Python packages for you to choose from, covering areas such as visualizations and graphics, data processing or analysis, statistic and numerical computing or machine learning and deep learning.

Now, the Mystic Pi package is used in many of the included notebooks. So let’s take a look and see how basically to access this notebook. Before hunting with notebooks, it is essential to basically understand the foundation, the foundation of Microsoft Sentinel, and that is the Log Analytics Data Store, which combines the high performance querying, dynamic schema and scales to massive data volumes. The Azure Portal and Microsoft Sentinel tools use a standard API to access this data store. The same API is also available for external tools such as Python or PowerShell. There are two main libraries that you can use to simplify API access and those are the KQL Magic library and the Mstic Pi.

Okay. The KQL magic. This library basically provides an easy to implement API wrapper to run KQL Queries. I’m not going to go deeper than this because it isn’t required in the certification exam, but you will find more information in the available downloadable resources for this particular lesson. The other one. The amstek pie. This. Microsoft intelligence. Microsoft threat intelligence. Python Security Tools is a set of Python tools intended to be used for security investigations and hunting. Many of the tools originated as code jupyter notebooks written to solve a problem as part of a security investigation. And some of the tools are only useful in notebooks. For example, much of the MBT Tools sub package, but many others can also be used from the Python command line or to be imported in your code.

The package addresses three central, let’s say, needs for security investigators and hunters acquiring and enriching data, analyzing the data and visualizing the data. The Amstek Pi can query using KQL. And the library also provides predefined queries for Microsoft Sentinel, microsoft 365 Defender for Endpoint, and the Microsoft Security graph. Again, you’ll have the examples in a link provided in the downloadable resources for this particular lesson. Now let’s take a look how we would create notebook. While, of course, we can run the Microsoft Sentinel notebooks in the Jupitery Lab or Jupyter Classic in Microsoft Sentinel notebooks are run on, as I mentioned, Asian machine learning on an Asia machine learning platform, to be more specific.

So to run notebooks in Microsoft Sentinel, you must have the appropriate access to both Microsoft Sentinel Workspace and an Azure Machine Learning Workspace. So the first thing you need to do in order to make use of the notebooks feature is to create an Asia Machine Learning Workspace from Microsoft Sentinel. And then the next thing is to actually launch a notebook in your Machine Learning Workspace. After you have created the Machine Learning Workspace over here, you can start launching your notebooks into Asia ML Workspace from Microsoft Sentinel. You can also view a notebook as a static document, such as in the GitHub built in Static Notebook Render. However, to run code in a notebook, you must attach the notebook to a back end process called the Jupitery Kernel.

Again, don’t worry, you’ll have additional documentation in the downloadable resources for this lesson. Now, let me quickly show you how you would do that. So, going here into our Microsoft Sentinel Workspace, let me just expand this a little bit. If we go to the notebooks blade over here, you can see that we have templates of notebooks which can be ran. But before that, you need to configure the Azure Machine Learning environment and you can do that from here. You can create a new Azure Machine. Learning workspace. Now, I am not going to show you how to do this because you will actually get to do this in the hands on lab at the end of the section.

So the last exercise from the lab will be exactly this to create a new Azure Machine Learning Workspace and to actually work with the first template over here with a notebook template. And again, you’ll have all the steps included in the lab. So, getting back to our slides, one last thing I want to mention, and that is you Can Also Explore The Notebook Code. So again, the following snippet here from the code is from the Getting Started guide for Microsoft Sentinel Notebook template and provides a representative example of working with Microsoft Sentinel data. So in the snippet of the Code we are basically here we are creating a new variable called Test query. The one over here. And this is mainly Python language. We are creating a new variable. Then we run our query over here and here.

This utilizes the Mstick Pi library to execute the KQL query in the Microsoft Sentinel log. Analytics related workspace. And the results are stored in the test DF variable over here. Again, the results, as you can see, of the query are stored in this variable. Now, next, we basically display the first five rows of the function. Now, I’m not going to go too much into it because this also involves a little bit of programming and working with Python and KQL at the same time. And with this Amsterdam Pi library. But I just wanted to quickly show you that you can also explore the notebook code and you can modify it, or you can amend it as per your requirements. Now, guys, this gets our lesson to an end our section to an end and the course to an end Again, please do the hands on lab available for this last section.

Also do the review questions, which are meant to test your knowledge on the topics that we discussed throughout this section. And as I’ve mentioned, this concludes our last lesson in the course. And this concludes the course. I hope you enjoyed the course, and I hope you found it to be valuable. And you got to learn lots of new and cool things. For the last time. I hope this has been informative for you. Good luck with your certification exam. If you are meaning to take that. And of course, good luck with all the security tools that we’ve discussed throughout the course and mainly Microsoft Sentinel that we’ve discussed for the last four sections. Again, I hope this has been informative for you and thank you for viewing.