It is time for part 8 of the Microsoft Defender for Endpoint (MDE) series. Part 8 is focused on the hunting experience in Microsoft 365 Defender. The advanced hunting feature and custom detection feature are part of the security.microsoft.com portal. Advanced hunting is based on the Kusto Query Language (KQL).

NOTE: The blog series focuses on features in Microsoft Defender for Endpoint P2 all Microsoft Defender for Endpoint P1 features are available in P2.

Specific question or content idea part of Defender for Endpoint? Use the contact submission form and share the post ideas.

Introduction

Defender for Endpoint is part of Microsoft 365 Defender. The portal contains different sections and data. Advanced hunting is a query-based tool that lets you explore all raw data. Each configured sensor sends telemetry and events directly to the Defender instance. Device events and alertinfo / alert evidence data is a directory available.

This part of the series is focussing on hunting with advanced hunting in Microsoft 365 Defender and the creation of custom detections.

Custom detections are powerful and can be used for improving the security response with automated device/ user actions and designing/creating more alerts for specific situations/ performed actions based on customer behavior or known blind spots based on MDE detection logic.

Advanced hunting basics

Advanced hunting is part of the security.microsoft.com portal and is based on the Kusto Query Language (KQL). KQL is an open-source language created by Microsoft to query big data sets stored in the Azure Cloud. The language is used in Log Analytics, Sentinel, Defender for Endpoint, Azure Data Explorer, and some more resources used in the Microsoft landscape.

Since most of the Microsoft 365 security products are visible in one unified portal it’s easier to search for data and perform unified hunting steps across all signals. advanced hunting is not only useful for threat/ security hunting. With advanced hunting the state of Defender can be easily validated for example:

  • Show all devices where MAPS is not configured
  • show all devices with outdated versions
  • Show all devices where the latest update is not installed
  • Show audit events of Attack Surface Reduction (ASR)
  • Show blocked network connections via SmartScreen

Important: Data from Microsoft Defender for Endpoint is retained for 180 days. In advanced hunting data is limited to max 30 days.

Advanced hunting is available in two modes; there is a guided and advanced mode. The guided mode is great when there is less experience with KQL. It is heavily recommended to start learning KQL – more on the learning topic later in this blog, there are great blogs/ resources and online guidances for learning KQL.

  • Advanced mode = Query in Editor
  • Guided mode = Query in builder

Microsoft information Microsoft: Kusto Query Language (KQL) overview

The query builder in guided mode enables a more visual for the creation of new queries without knowing the Kusto Query Language(KQL). Each tier of experience can use the query builder and filter the data. It is recommended to start learning KQL and the more advanced editor. KQL is so powerful and more needed in the future for creating detections and hunting.

Tip; open the advanced query with the button; Edit in KQL. Viewing the query in KQL is useful for learning the language from UI input and learning the generated KQL.

Data Sources

Currently, the following applications/ services are part of the advanced hunting platform in Microsoft 365 Defender/ security.microsoft.com:

  • Microsoft Defender for Endpoint (MDE)
  • Microsoft Defender for Cloud Apps (MDA)
  • Microsoft Defender for Office (MDO)
  • Microsoft Defender for Identity (MDI)
  • Azure AD Identity Protection (AADIP)
  • Azure AD user/ sign-in information (AAD)

Schema mapping:

Mapping from main product to schema name:

MDEMDAMDOMDIAAD IPAADAlerts
Devices/ TVMApps & IdentitiesEmail and CollaborationApps & IdentitiesApps & IdentitiesApps & IdentitiesAlerts

For the full schema reference/explanation see: Data tables in the Microsoft 365 Defender advanced hunting schema | Microsoft Learn

  • Schema name: Devices/ Apps & Identities/ Email and Collaboration/ TVM
  • Schema tables: DeviceInfo/ AlertInfo/ EmailEvents

Known limitations in Advanced Hunting

Currently, there are a couple of known limitations/ service limits for keeping the service operational.

Data range: Each query can look up data from up to the past 30 days. Data timeframe is the same for hunting and custom detections

Result set: Each query can return up to 10.000 records. Use always filtering where possible to define results

Max timeout: Each query can run for up to 10 minutes. When not completed in 10 minutes there will be a service error

CPU resources: When queries are run over 10% of the allocated resources, there will be an error. Queries are blocked if the tenant has reached 100% until after the next 15-minute cycle. Always optimize queries.

Defender Portal

Advanced hunting is part of the Microsoft 365 Defender and is available via “Hunting”. The hunting experience created in the portal is for all sources explained above. When opening the advanced hunting view the main view is visible:

Let’s start with the top bar explanation in advanced hunting. Currently, the following items are visible in the top bar:

  • Help resources: Redirect to known sources for advanced hunting and KQL
  • Query resources report: Report for checking the used resources
  • Schema reference: Search for all schemas and view additional information and connected columns.

The main section shows 4 components:

  • Schema: Available data schema for each product/ section.
  • Functions: Created functions
  • Queries: Saved queries and community queries ( with the option for sharing queries with the team) or save queries for own usage
  • Detection Rules: Detection Rules overview

The middle section of the screen is focussing on the actual hunting experience where KQL queries can be created.

In the query section, new queries can be created. Results show all results up to a max of 10.000 results:

Advanced hunting – find heavy queries or auditing

Common question; can we see all queries performed by the security team? The answer is yes. Part of advanced hunting is the Query resources report. With the use of the query resource report, it is possible to find queries with the highest amount of CPU resources and running time. The same view can be used for auditing the running queries.

Where to find Query resource report? advanced hunting -> right top “Query resource report”

Each member with Defender for Endpoint permissions can view all their own queries. Only the AAD global admin, AAD security admin, and AAD security reader roles can see queries from all users in all dashboards and reports.

The report table default displays queries from the last day and is sorted based on resource usage. Additional filters can be used for longer timeframes or filtering for specific interfaces/ resource usage.

Tip; When opening the resource report it is possible to view old queries for up to 30 days and open the query directly in the query editor. Useful when the query is broken after changes and unsaved, or the query is lost without saving.

Advanced hunting – Sample queries and shared queries

Each query can be saved in the queries section of advanced hunting. Press the Save button and fill in the query name and location.

  • My queries: Only visible for the signed-in user
  • Shared queries: Visible for all users with Microsoft 365 Defender access

When needed, create a new folder for bringing some structure to the My queries or Shared queries.

After saving – all queries are visible in the Queries section and can be easily re-used/ edited.

Custom detection rules

One of my favorite features in Defender for Endpoint is the option for creating detection rules based on advanced hunting queries. Each environment requires custom detections for specific apps, behaviors, and more. Microsoft is not sending alerts for each “potentially malicious” activity, there is way more data available directly in Defender for creating custom detection rules. Typical custom detection rules:

  • A new local user-created
  • SmartScreen override
  • Forbidden software installed
  • Remote tools installed and connected outbound (AnyDesk, TeamViewer)
  • Local account created
  • IoC matched against public CSV
  • Launch of an executable or script from C:\Users\Public\ or any other location
  • Run cipher.exe on the system

What is it?

Custom detection rules are rules which make it possible to proactively monitor various events and system states. Each detection rule can be configured to run at regular intervals and generating alerts and taking “automated” response actions when there is a match.

Before we start with custom detection rules it is good to know rules can be created with the Security administrator/ Security operator roles or custom roles in MDE.

Important: When the advanced model is enabled in Defender for Endpoint it is required to assign the manage security setting permissions in MDE for security operators.

Custom detection rules contain a various list of needed fields. For each detection rule it is needed to get the following result:

  • Timestamp
  • ReportId
  • Accepted columns that identify specific devices, users, or mailboxes (source)
Important: To prevent the service from returning too many alerts, each rule is limited to generating only 100 alerts whenever it runs.

Why do we need proactive hunting?

Not all threat scenarios begin with an alert. With the use of proactive hunting, there is way more value possible instead of only protection. For each organization the use cases are different. With the use of advanced hunting, there is the option for getting more information on entities and available IOCs. For example; show me all activity for website xx, or show all activity from a specific IP address.

How to create detection rules?

Create the KQL query in the query editor and select Create detection rule to specify all required information including the general details. Tip; create a clear and short title with the most information. Don’t use alert titles such as “Custom Detection 1”

Fill in all required information. Each rule checks for matches from the past 30 days. The rule can be created with multiple frequency intervals. The following schema is available:

  • Every 24 hours—runs every 24 hours, checking data from the past 30 days
  • Every 12 hours—runs every 12 hours, checking data from the past 48 hours
  • Every 3 hours—runs every 3 hours, checking data from the past 12 hours
  • Every hour—runs hourly, checking data from the past 4 hours

Recently Microsoft announced the availability of the Continuous NRT frequency rules. With the NRT rules it is possible to generate near real-time new custom detections and identity threat faster.

Currently, NRT is supported for queries when there is only one table in use and there is no join, unions, or external data operator part of the query. Currently NRT is supported for the following tables:

  • AlertEvidence
  • DeviceEvents
  • DeviceFileCertificateInfo
  • DeviceFileEvents
  • DeviceImageLoadEvents
  • DeviceLogonEvents
  • DeviceNetworkEvents
  • DeviceNetworkInfo
  • DeviceInfo
  • DeviceProcessEvents
  • DeviceRegistryEvents
  • EmailAttachmentInfo
  • EmailEvents
  • EmailPostDeliveryEvents
  • EmailUrlInfo
  • UrlClickEvents

More information for NRT rules: Continuous (NRT) frequency

Impacted entities

Impacted entities are important to query and configure the correct impacted entities. The impacted entity is the device, user, or impacted entity. Device mapping is for example possible with the DeviceId or DeviceName. User is possible with the AccountSid. The wizard shows only the columns that are returned by the query.

Actions

Each custom detection can automatically take response actions on devices, users, files, or emails. For devices, it is possible to isolate, collect investigation packages, run antivirus scans, initiate investigations, or restrict app execution.

For files, Users, and Emails multiple actions are available. There is some mapping with recommended fields. For the category of each specific action (Device, User, Email, File) there are tables needed. For example; for Device actions, the DeviceId table is required, and for User actions, one of the tables (AccountObjectID, InitiatingProcessAccountObjectId, RecipientObjectId) is required.

For email actions, both NetworkMessageId and RecipientEmailAddress must be available in the result.

Column Device actionsUser actionsEmail actionsFile actions
DeviceId✅❌❌❌
SHA1❌❌❌✅
SHA265❌❌❌✅
InitiatingProcessSHA1❌❌❌✅
InitiatingProcessSHA256❌❌❌✅
AccountObjectId❌✅❌❌
InitiatingProcessAccountObjectId❌✅❌❌
RecipientObjectId❌✅❌❌
NetnworkMessageId❌❌✅❌
RecipientEmailAddress❌❌✅❌

How to view detection rules?

To view all created detection rules, go to Hunting -> Custom detection rules and open the specific rule for all information. In this view, all information is available including the last run, last run status, frequency, next run, and updated time.

Click on the open detection rule page for showing all related information, including alerts

Detection rule page shows all triggered actions and triggered actions part of the rule.

How to learn KQL?

Of course; we can explain in this blog post the basics of KQL and share multiple queries, all there are better resources for KQL. First, we start what is by default available in the Microsoft Defender 365 portal.

Threat analytics

The MDE threat analytics reports are really useful for learning KQL. Some of the threat reports contain detailed KQL queries. For example; IRIDIUM uses TOR hidden services on targets for persistence and evasion the report contains multiple KQL queries.

Microsoft resources

Community training

Learning KQL? the Must Learn KQL training created by Rod Trent is a must-follow. The training is advanced and perfect for learning KQL.

Community contributions

Of course; the community shares lots of KQL-related information. Matt Zorich published a large set of queries that can be used for inspiration. Gianni shares quality content on the KustoKing.com website and the KQLCafe which is interactively and hosted on YouTube. The below list contains some of the community contributions (missing someone?; please ping me)

Sources

Microsoft: Create and manage custom detections rules

Microsoft: Advanced hunting overview

Part 8 of the Microsoft Defender for Endpoint series is completed – focused on the explanation of the advanced hunting/ KQL/ custom detection features.

Searching for specific Defender for Endpoint information? Use the contact submission form and share the post ideas or contact using Linkedin or Twitter. I will take all suggestions into the Defender for Endpoint series and help the community as far as possible.

View previous part – Microsoft Defender for Endpoint series – integrations with other products – Part7 

View next part – Automation via Logic Apps and Sentinel – Part9