For years, customers have asked for ways to extend data retention in Microsoft Defender XDR beyond the default limits to support advanced hunting and long-term archiving needs.
By default, Defender XDR retains incidents, alerts, and related data for up to 180 days. However, Advanced Hunting data is only available for 30 days. For proactive threat hunting, incident response, and compliance-driven investigations, these retention periods are often insufficient. Extended data storage is essential to meet regulatory requirements and enable meaningful long-term analysis.
Many organizations need to keep security data accessible for longer than three months. In some cases, this is driven by compliance and regulatory mandates. In others, security teams need historical data to investigate threats, identify trends, or perform retrospective hunting. A 30-day investigation window is simply too short. Effective security operations require access to a much broader data set over time.
Since last week Microsoft released the general availability of data lake tier ingestion for Microsoft XDR Advanced Hunting tables directly into Microsoft Sentinel data lake. With data lake tier ingestion, Sentinel data lake becomes a must-have destination for XDR insights, enabling users to:
- Store high-volume Defender Advanced Hunting data efficiently at scale
- Extended security analysts’ lifespans for investigation/ compliance and threat research
- Query data using KQL explorer, KQL Jobs, and Notebook Jobs
- Integrate via MCP server
- Threat hunting with custom Sentinel graphs
- Reduce Sentinel ingestion cost
Workaround (the “old clever solution”)
Some months ago, I published the following blog: How to store Defender XDR data for years in Sentinel data lake without expensive ingestion cost
To avoid Sentinel ingestion costs, the following was possible:
- Workspace transformation rules
- Custom tables set to the data lake tier
- Copying Defender XDR data into those tables
This worked, but it was:
- A little complex to design
- Harder to maintain
- Still felt like a hack around platform limitations
The difficulty wasn’t technical skill – it was that the platform didn’t natively support direct low-cost storage for Defender XDR data. Microsoft has since introduced table-level tier management for Sentinel, including Microsoft-managed security tables.
What is new:
- Directly switch Defender XDR tables to the Data lake tier
- No workspace transforms needed
- No custom tables needed
- No duplicate data
Which options are available?
Previously, there were a couple of options with the use of the streaming API in Defender XDR and Azure Data Explorer:
- Azure Data Explorer
- Stream data directly to Log Analytics
- Stream data directly to a Storage account/ event hub
Azure Data Explorer
In the past years, I deployed many Azure Data Explorer clusters for smaller companies and larger enterprises with up to 6TB ingested daily. Previously, it was quite hard to have a solution where the KQL language is available at a low cost. Azure Data Explorer is a great solution without relying on Microsoft Sentinel for the expensive Analytics tier ingestion cost.
The downside of Azure Data Explorer is the maintenance and complexity of setting up a cluster. In short, it requires:
- Eventhub or Storage account
- Maintain Throughput Units to handle the events per second to avoid data loss
- Azure Data Explorer cluster
- Azure Data Explorer sizing
- Need to maintain and track performance/ ingestion
- Need to be parsed and transformed
- Tables need to be maintained
- Active reporting to track performance peaks
- Maintenance for new tables and data schemas
- And many more
So keep in mind that running Event Hub Namespaces and Azure Data Explorer clusters requires additional troubleshooting/ operational tasks and active reporting. When the people and knowledge are available, it is a great solution for relatively low cost – it seems still cheaper in comparison with the new Sentinel data lake, all keep in mind – it requires operation and more maintenance to keep it running. It is far from managed.
Log Analytics/ Microsoft Sentinel
Streaming logs directly in Log Analytics/ Microsoft Sentinel is the easiest way – and also the most expensive way of ingesting and storing data for archiving purposes. You’ll be billed for the ingestion into Sentinel before these logs can be stored. And ingestion in Microsoft Sentinel is not cheap. The benefit is that Microsoft is automatically creating the mappings and doing all the data transformations as part of the ingestion.
Downside: for 1TB a day in the Analytics tier, it costs only for the ingestion without any retention $3207 a day. That is over $95.000 for 30TB of ingestion each month.
Without any discount or reservation tier. Based on the pay-as-you-go ingestion cost. All still with additional discount or reservation tiers, it is still expensive for just storing data longer.
Why is it a challenge?
Advanced Hunting is included in Defender XDR. This data is part of the Defender license and is already paid for. All hunting queries and detections can be created directly on top of the Defender XDR dataset. With the migration from Sentinel to the Unified SecOps platform, it is increasingly clear that Microsoft is moving toward centralizing analytics and detections within Defender XDR.
As a result, the data is available in Defender XDR and then mirrored to Sentinel, where customers pay Sentinel’s relatively high data ingestion costs. This approach is not always optimal, as the same outcomes can often be achieved more efficiently without ingesting the data into Sentinel.
Using storage accounts or Event Hubs may sound like a good alternative, but there is a significant drawback: the KQL experience is not available. As a result, searching and recovering data can be difficult and time-consuming.
Solution we need
Personally, the end solution is quite straightforward and aligns with Microsoft’s overall strategy: use the Defender XDR dataset for running queries, analytics, and all detection-related functionality, while storing the data cheaply for longer retention without the need to ingest it into Sentinel.
The new solution: Microsoft Sentinel data lake
Microsoft Sentinel data lake is the new solution, which is cost-efficient and a way to store data for years as part of the data lake. Sentinel Data Lake was announced on July 22nd 2025, marking a new chapter for log management in the Microsoft security ecosystem.
Microsoft says the following:
“Sentinel data lake simplifies security data management, eliminates security data silos, and enables cost-effective long-term security data retention with the ability to run multiple forms of analytics on a single copy of that data”.
In short, it means that the Sentinel data lake is a managed data lake and is managed by Microsoft. Microsoft is maintaining and scaling the backend and making sure the performance is scalable and efficient for the amount of data.
Some months ago, I published the following blog; How to store Defender XDR data for years in Sentinel data lake without expensive ingestion cost where I explained a solution to transform logs from Sentinel using DCR transformations and store them in a custom table.
The good news is that Microsoft has changed this behavior and now allows Defender XDR tables to be moved directly. This means it is no longer necessary to create custom tables or publish and maintain DCR transformation rules to process the data.
This was some months ago. No option to store logs directly in the data lake tier:
With 30-day analytics (as part of Defender XDR) and 1-year total retention, it will perform the following:
- In the Sentinel Defender XDR connector, the DeviceProcessEvents will be streamed to the Log Analytics Workspace (LAW)/Sentinel
- Data will first go to Analytics, meaning it will be ingested into Sentinel Log Analytics and become part of the Analytics tier before it is sent to the data lake)
Conclusion: Additional cost for the ingestion, and still based on Microsoft Sentinel ingestion as part of the connector. Not ideally, since the data is ingested in the expensive Analytics tier first. The “included in license” is focused on the Defender XDR Advanced Hunting data.
This is not the preferred approach. Recently, Microsoft made it possible to switch Defender XDR tables directly to the Data Lake tier (Lake-only ingestion). This new capability allows you to store tables in the Data lake tier without first ingesting them into the Analytics tier or configuring complex filtering rules.
Let’s explain this more in-depth:
Switch table to data lake only tier
To view and manage table settings in the Microsoft Defender portal, follow these steps: Select Microsoft Sentinel > Configuration > Tables. Now select one of the Defender tables (only tables as part of the Defender XDR connector can be switched), for example the DeviceProcessEvents or DeviceEvents table.
Data lake tier: Set Retention to a value between 30 days and 12 years. Selecting Data lake tier stores data exclusively in the data lake. When the tier is changed to data lake only; the tier table is only storing data in the data lake tier. Which means there will no data be ingested in the Sentinel Analytics tier. Please be aware since not all the features are available in the data lake tier, such as; alerting, analytics rules and custom detection rules. All this is in the most cases fine, since Defender XDR data is already part of Defender Advanced Hunting for 30 days where you can build automation, playbooks, and detections.
For switching to the Data lake tier; select the Data lake tier and specify the retention:
So be aware, before switching
- Check that all detections have been migrated to Defender XDR
- Check that playbooks and automation rules have been transformed or migrated to Defender XDR
- Check that other items, such as workbooks and dashboards, have been migrated to Defender XDR
- etc. etc.
Data will be streamed directly from Defender XDR to the Sentinel data lake. without first being ingested into the Analytics tier.
When data is only available in Sentinel Analytics for archiving purposes, you are good to go to switch the data directly to Defender data lake. All better check before some functions will break. On the backend, the table is switched from Analytics to a Custom table with the Auxiliary/Data Lake plan.
With the following configuration, where DeviceEvents is configured as Data lake tier only. The data is stored for 1 year (up to 12 years) in Sentinel data lake. 30 days of data are still available in Advanced Hunting for detections and hunting.
The following tables are currently supported:
- DeviceInfo
- DeviceNetworkInfo
- DeviceProcessEvents
- DeviceNetworkEvents
- DeviceFileEvents
- DeviceRegistryEvents
- DeviceLogonEvents
- DeviceImageLoadEvents
- DeviceEvents
- DeviceFileCertificateInfo
- EmailEvents
- EmailUrlInfo
- EmailAttachmentInfo
- EmailPostDeliveryEvents
- UrlClickEvents
- CloudAppEvents
- IdentityLogonEvents
- IdentityQueryEvents
- IdentityDirectoryEvents
- AlertInfo
- AlertEvidence
All of the tables are part of the Defender XDR data connector in Sentinel. Currently, all tables with the table type “XDR” are not supported. Examples include IdentityInfo, BehaviorInfo, and other XDR-related tables. Those XDR tables are also not (yet) supported for Sentinel data lake.
Conclusion
Microsoft Sentinel provides a data lake designed for cost-efficient, long-term storage of security data. The Sentinel data lake allows organizations to retain large volumes of logs without paying continuous ingestion and analytics costs, making it a strong option for compliance-driven retention and historical investigations.
However, while the data is stored in the Sentinel data lake, it is not intended for day-to-day detection engineering or threat hunting purposes. The full KQL experience, low-latency queries, and native security context are available in Microsoft Defender XDR, where analytics, detections, and hunting queries are meant to be authored and executed. This aligns with Microsoft’s Unified SecOps strategy, which positions Defender XDR as the primary analytics and hunting layer, with Sentinel acting as the long-term storage and SIEM backbone.
In practice, this means security teams should run active investigations and hunting queries in Defender XDR, while relying on the Sentinel data lake for extended retention, historical lookbacks, and compliance needs. This separation enables better performance, lower costs, and a clearer operational model for modern SOCs.
With the use of Sentinel data lake in combination with Defender XDR, you can save costs and configure the data storage environment more efficiently, without the need to configure third-party tooling for longer retention periods.
More information about Sentinel data lake: Microsoft Sentinel data lake: How to use/enable and set-up the unified data lake
Sources
Microsoft Sentinel data lake overview – Microsoft Security | Microsoft Learn
Data lake tier Ingestion for Microsoft Defender Advanced Hunting Tables is Now Generally Available
