By default Microsoft Defender for Endpoint stores Endpoint events in Defender for Endpoint for the configured retention period; Max: 180 days.  For longer data retention it is possible to export events to external sources, most typical; Azure Storage or Azure Event Hubs.

In this blog the explanation and the usage of the new Defender streaming API Public Preview functionality for exporting “specific “events to your storage account/ event hub. From the Azure Storage or Event Hub it is possible to export data into other SIEM solutions.

The Streaming API exports the selected event types in the Microsoft 365 Defender Advanced Hunting schema.

NOTE: Feature currently in Public Preview

Announcement Microsoft

Requirements

  • Azure Storage Account
  • Azure Event hub
  • Resource Provider: Microsoft.Insights enabled
  • Defender for Endpoint instance

Defender 365 Streaming events

Microsoft 365 Defender Streaming API lets you export events to your Azure Event Hubs or your Azure Storage account. The current Streaming API preview contains the following events types (based on the Advanced Hunting Event table name)

Alerts: AlertInfo, AlertEvidence

Devices: DeviceInfo, DeviceNetworkInfo, DeviceProcessEvents, DeviceFileEvents, DeviceNetworkEvents, DeviceRegistryEvents, DeviceLogonEvents, DeviceImageLoadEvents, Device Events, DeviceFileCertificateInfo

Email: EmailEvents, EmailAttachmentInfo, EmailUrlInfo, EmailPostDeliveryEvents

In comparison with the other Defender for Endpoint RAW data functionality gives the Streaming API more options for Alerts en Email alerts.

With the streaming API, it is possible to export the selected events to an Azure Event Hub or Azure Storage account.

The following forwarding systems are available from the Streaming API settings:

  • Azure Event Hub
  • Azure Storage account

For opening the new streaming API settings.

  1. Go to Microsoft 365 security portal
  2. Open settings
  3. Open Microsoft 365 Defender
  4. Open the tab Streaming API

Create Azure Event hub

For exporting Security events into the Azure event hub it is required to create first an Event hub in your tenant. For creating the event hub:

Register Resource provider Microsoft.Insights

For the Azure Event Hub the Microsoft.Insights resource provider is required. Go to the subscription settings and register the Microsoft.Insights Resource Provider.

  1.  Go to Subscriptions > Your subscription > Resource Providers > Register to Microsoft.Insights.
  2. Create Event Hub Namespace and fill in the required Namespace name, location, and throughput units.

For other technical instructions about the Storage Account creation/ networking and permissions. Read the technical docs from Microsoft.

Export events

The next step is based on the Streaming API configuration. For configuring the streaming API settings:

  1. Go to Security.microsoft.com -> Settings -> Microsoft 365 Defender
  2. Select the setting Streaming API
  3. Click Add
  4. Fill in the name (1)
  5. Select the option Event Hub or Azure Storage (2)
  6. Select the event types for exporting to Azure storage or Azure Event Hub (3)

Azure storage configuration

When using the Azure Storage configuration option. Select the check-box: Forward events to Azure Storage and fill in the Storage Account Resource ID.

For getting the storage account resource ID value:

  1. Open the Storage Account resource
  2. Go to settings -> Endpoints
  3. Find the value Storage account resource ID

Event hub configuration

For the Event hub configuration. Select the check-box: Forward events to Event hub. And configure the Event-Hub Resource ID & Event-Hub name.

For getting the Event Hub name and Resource ID value:

  1. Open the Event Hub
  2. Go to Properties
  3. Find the value Name  & Resource ID

Select events

From the Streaming API creation page – select the events for the specific streaming task. With this, you can select only the events needed and reduce the cost for long-term storage in case of specific events.

Azure Storage account result

The blog container will be created and filled in with all the configured event data. Each advanced hunting event type will be created in a separate blob container. The result for all the selected events:

The schema of each row is based on the following structure:

{
        "time": "<The time WDATP received the event>"
        "tenantId": "<Your tenant ID>"
        "category": "<The Advanced Hunting table name with 'AdvancedHunting-' prefix>"
        "properties": { <WDATP Advanced Hunting event as Json> }
}

A blob container will be created for each event type:

An example of an incident event:

Location: Y=2021/ m=06 /d=07 /h=19 /m=00. Details included in PT1H.json file.

Azure Event Hub results

The schema of each row is based on the following structure:

{ "records": [ { "time": "<The time Microsoft 365 Defender received the event>" "tenantId": "<The Id of the tenant that the event belongs to>" "category": "<The Advanced Hunting table name with 'AdvancedHunting-' prefix>" "properties": { <Microsoft 365 Defender Advanced Hunting event as Json> } } ... ] }

Mapping data

For mapping the data values it is important to map the data types for each event. From the advanced hunting query functionality, it is possible to get the data types mapping for each event from the AdvancedHunting structure. See Advanced hunting reference for detailed info

Example for getting the data types for the DeviceInfo event: replace DeviceInfo value for other event types

Sources

Microsoft: Announcing Microsoft 365 Defender Streaming API Public Preview – Microsoft Tech Community

Microsoft: Stream Microsoft 365 Defender events | Microsoft Docs