How to troubleshoot Live Response in Defender for Endpoint

Live Response is a powerful feature as part of the Microsoft 365 Defender portal. With the use of Live Response Security Operations Teams can establish a remote session to collect more files/ or forensic evidence/ run scripts remotely.

With the use of Live Response, it is possible to run a couple of pre-defined commands, and even more powerful is the ability to upload custom PowerShell scripts. In short Live Response is a cloud-based interactive shell that is fully integrated in the Defender for Endpoint product.

What is the challenge?

In the past years, I deployed Defender for Endpoint in many large environments and discovered sometimes “issues/limitations” related to the Live Response capability. Common examples;

• Session stuck
• Session slow/ Commands slow
• Timeout
• File not collected
• Download error
• Index error
• No result or status update
• History of live response commands
• Delay in Live Response connectivity

In this blog, I will explain the troubleshooting capabilities more in-depth with a strong focus on situations where Live Response is not working as expected.

---

How works Live Response?

The live response capability gives security operations teams instantaneous access to a device using a remote shell connection which is included in the portal. This feature has been integrated into the Microsoft 365 Defender Portal and can be enabled from the Advanced Features blade. For Live response, there are a couple of settings:

• Live response
• Live Response for Servers
• Live Response unsigned script execution

Live response service components

First, let’s explain some of the key services as part of the Defender for Endpoint component:

• MsSense.exe: This process is the main service executable for Defender for Endpoint. The service is running as a service with the name SENSE.
• SenseIR.exe: This executable spawns as a child process of MsSense.exe when initiating a Live Response Session via the Microsoft 365 Defender portal.

Live response leverages the WNS service in Windows. The Windows Push Notifications System Service (WNS Service) interacts with the WNS cloud service to initiate the live response connection.

Based on the above information the following two services are involved: MsSense.exe and SenseIR.exe

MsSense.exe is always active when the machine is correctly onboarded to Defender for Endpoint and will start the SenseIR.exe process as a child process when the live response connection is initiated – most of the Live Response actions will be performed via the SenseIR.exe process.

Process Explorer; without Live Response active

Process Explorer; with Live Response session active, the sub-process SenseIR is visible as part of the MSSense service.

Another key component of Live Response is the centralized script library. Each script will be hosted in a centralized tenant library and is available for all users and machines. There is no limitation based on machines or device groups for the centralized library. Good to know – the library is part of the MDE instance, and not shared with other tenants.

Initiating Live Response session

During the initiating of a Live Response session, it will show the following output in the command line interface as part of Microsoft 365 Defender:

“Session established”

When the “session established” message is visible – it means the SenseIR component/ executable is started and registered via the MsSense.exe service. When the Live Response session is not running/ active the SenseIR.exe process is not active/ running.

Each unique Live Response session is generating a unique Session ID. This ID is used for auditing/ tracking and additional operation actions (more on this later). The Session ID is visible in the left panel of the Live Response session.

When the session is still trying to initiate it is showing the status “connect” with the duration time in the left entity summary view:

Initiating Live Response session

As already mentioned the Live Response commands can be extended with custom uploaded PowerShell scripts. Each file needs to be uploaded to the centralized library. This is not an interactive PowerShell session; in general, there is another child process of SenseIR.exe for the PowerShell.exe and when needed additional services.

When running the PowerShell script directly it will result in a new PowerShell child process of SenseIR.exe.

When we check the process details there is more information visible. The command line is:

C:\Windows\system32\WindowsPowerShell\v1.0\powershell.exe -ExecutionPolicy Bypass -NoProfile -NonInteractive -Command "& {$OutputEncoding = [Console]::OutputEncoding =[System.Text.Encoding]::UTF8;$scriptFileStream = [System.IO.File]::Open('C:\ProgramData\Microsoft\Windows Defender Advanced Threat Protection\Downloads\PSScript_{2A02016D-6C6F-4E15-AC24-84A9D924C00C}.ps1', [System.IO.FileMode]::Open, [System.IO.FileAccess]::Read, [System.IO.FileShare]::Read);$calculatedHash = Get-FileHash 'C:\ProgramData\Microsoft\Windows Defender Advanced Threat Protection\Downloads\PSScript_{2A02016D-6C6F-4E15-AC24-84A9D924C00C}.ps1' -Algorithm SHA256;if (!($calculatedHash.Hash -eq '67aac04037cf0da7090448d7f0668538c2dd4c1dc845d1e47f1fee84886a1bda')) { exit 323;};Start-Transcript -Path 'C:\ProgramData\Microsoft\Windows Defender Advanced Threat Protection\Temp\PSScriptOutputs\PSScript_Transcript_{2A02016D-6C6F-4E15-AC24-84A9D924C00C}.txt'; . 'C:\ProgramData\Microsoft\Windows Defender Advanced Threat Protection\Downloads\PSScript_{2A02016D-6C6F-4E15-AC24-84A9D924C00C}.ps1' }"

When the script is performed the uploaded library script will be downloaded in the C:\ProgramData\Microsoft\Windows Defender Advanced Threat Protection\Downloads\PSScript_ location. This location is used as a temp store for the file. Each script included a unique hash which is used for the script download and temp location.

Two important locations

Temp location of the PowerShell script:

C:\ProgramData\Microsoft\Windows Defender Advanced Threat Protection\Downloads\PSScript_{2A02016D-6C6F-4E15-AC24-84A9D924C00C}.ps1

PowerShell Transcript output:

C:\ProgramData\Microsoft\Windows Defender Advanced Threat Protection\Temp\PSScriptOutputs\PSScript_Transcript_{2A02016D-6C6F-4E15-AC24-84A9D924C00C}.txt

When the script is completed it will show the Transcript started message and it will also pipe all content into a file. This redirects to the same temp directory output and contains the PowerShell transcript information.

Once the script execution is finished, the output is printed on the console. For example; when running Get-MpComputerStatus and Get-MpPreference via custom PowerShell script the following output is visible directly in the Live Response output.

Known limitations

The Live Response service has some limitations when running specification actions. The following limitations are applicable when using Live Response:

• 25 live response sessions at a time
• Inactive timeout of a session is 30 minutes
• Individual live response commands have a time limit of 10
• Getfile, findfile and run have a limit of 30 minutes
• One single user can initiate 10 concurrent sessions
• A device can only be in one session

Be aware when running large scripts/ searches/ applications – there will be a potential timeout after 30 minutes when the session is started.

The following file size limits are applicable:

• getfile limit: 3 GB
• fileinfo limit: 30 GB
• library limit: 250 MB

---

Troubleshooting

PowerShell script not available?

The PowerShell script must be always located in the centralized library. When the script is uploaded it will take a couple of times before the script is uploaded. (up to 10 minutes, most of the time within a couple of minutes).

When the file is not found in the library; the following message is visible: “Error: File ‘xxx’ wasn’t found in the library. Run ‘library’ to get list of files”

The content of the library can be validated via the Live Response session and API:

Live Response

Run the command Library as part of the active Live Response session. It will show all files as part of the centralized library.

API

Via the API explorer part of Microsoft 365 Defender, it is possible to check the library content.

https://api.securitycenter.microsoft.com/api/libraryfiles

With the use of the library files API, the uploaded files are directly visible, including additional information.

Unsigned scripts

For unsigned scripts, the advanced feature Live Response unsigned script execution must be enabled. When not enabled Live Response is not allowed to run unsigned scripts. Enablement is available directly via Advanced Features in Microsoft 365 Defender – Endpoints.

No connection or delay during initial connections

When the live response session cannot be initiated it is good to validate the service statuses. SENSE must be always in a running state; when SENSE is not running it will not work. Live response leverages Defender for Endpoint sensor registration with WNS service in Windows.

In situations where it is not possible to set up a live response session or the session is slow, confirm the following:

• WpnService (Windows Push Notifications System Service) isn’t disabled
• WpnService connectivity with WNS cloud isn’t disabled via group policy or MDM setting.

Based on experience in some environments the WpnService connectivity is disabled by GPO; since this is part of some of the security baselines. More information: Slow live response sessions or delays during initial connections

Live response/ WNS is using the following endpoint URL. When Live Response is slow, check if the following URLs are available. For the best performance, a Direct Connection/ Proxy bypass is recommended. More information:

Endpoint/URL	Description
*.wns.windows.com	Windows Push Notification Services (WNS) – Live Response
login.microsoftonline.com	Windows Push Notification Services (WNS) – Live Response / Vulnerability assessment for network devices / Security Management for Microsoft Defender for Endpoint – Azure Registration
login.live.com	Windows Push Notification Services (WNS) – Live Response

View session status (failed/ Completed)

When using Live Response via automation (Logic Apps/ Sentinel or other automation platforms) it is useful to view the status. Live Response sessions will be reported directly in the Microsoft 365 Defender Action Center view. When going to Microsoft 365 Defender -> Action Center -History all actions will be visible.

Live Response sessions will be visible under the Action type: Live Response command. Check the status of the Live Respons command is initiated correctly. For actions performed by automation, it is useful to filter on “submitted by” name.

When opening the investigate ID it will show more information and the actually live response command console. In the below situation the error “Unexpected error. HRResult: -2145844840” is visible, most of the time related to the activity of the machine. (offline/ no network activity).

For auditing it will show all performed commands as part of the live response session:

View via API

The machine action API can be used for checking the status of the Live Response session. In some situations when running from automation (Palo Alto XSoar/ Logic Apps/ API) the session is not directly visible in the Action Center. With the API it is possible to receive all machine actions. When needed add some filters as part of the API call.

https://api.securitycenter.microsoft.com/api/machineactions

---

Troubleshooting automation

When running Live Response Sessions by automation it is a bit more difficult; since there is no full diagnostic trace for the Live Response Session.

When building automation there are a couple of steps part of the automation:

1. Run a new live response session
2. Check for session rules
3. Get the results

The live response is initiated with the machine ID of the machine + body input where the action schema is defined:

https://api.securitycenter.microsoft.com/api/machines/{machine_id}/runliveresponse 

Part of the body request:

{
"Commands":[
      {
         "type":"RunScript",
         "params":[
            {
               "key":"ScriptName",
               "value":"forensics.ps1"
            }
         ]
      },
      {
         "type":"GetFile",
         "params":[
            {
               "key":"Path",
               "value":"C:\\windows\\Temp\\Services.csv"
            }
         ]
      }
   ],
   "Comment":"Live Response API Forensics collection test"
}

Part of the request is the status; when the action is initiated via API it will be always with the status Pending. The id part of the response is the Live Response Action ID. This machine ID can be used via the machine action API.

Action already in progress

When there is already an action in progress there will be an error with the code “ActiveRequestAlreadyExists”.

Live Response supports only one active action as part of the machine. Multiple Live Response actions are not supported. Part of the error is the active action ID:

Track status

With the machine actions API, it is possible to track the status of the Live Response action.

https://api.securitycenter.microsoft.com/api/machineactions/{machine_action_id} 

Part of the result is the status of the Live Response session. The status is now “succeeded” and the action is updated with all information.

Part of the commands is the RunScript and GetFile actions. Both actions are completed. When the action is failed it will be visible in the command status/ errors field.

In the above screenshot, there is the index value 0-1 visible. This index is the order/ location of the action and can be used to download the output. When running the following command it is possible to download the output or downloaded file.

https://api.securitycenter.microsoft.com/api/machineactions/{machine_action_id}/GetLiveResponseResultDownloadLink(index=1) 

If a command has ended successfully, the response will include a blob storage link to the script output (for RunScript) or collected file (for GetFile). The download link will expire within 30 minutes. The correct Index value can be checked with the machine actions API.

Machine offline?

If the machine isn’t available; the sessions are queued for up to three days when using the API. During the next active timestamp, it will be performed and queued on the backend.

Cancel automation actions

When needed – the action can be canceled via the following API call:

https://api.securitycenter.microsoft.com/api/machineactions/<machineactionid>/cancel

API limitations

• 10 calls per minute rate limitate
• 25 concurrently running sessions
• RunScript timeout after 10 minutes
• Live response commands can’t be queued up and can only be executed one at a time.
• Multiple live response commands can be run on a single API call. However, when a live response command fails all the subsequent actions won’t be executed.
• When RBAC grouping is enabled the automated remediation level must be assigned, at least with a minimum Remediation Level
• Multiple live response sessions can’t be executed on the same machine

More information: Live Response API Limiations | Microsoft

---

Troubleshooting via event viewer

Part of the event viewer is the SenseIR event log. Event viewer -> Application and Services logs -> SenseIR -> operational

Part of the event viewer is more in-depth visibility in the output and performed actions from the SenseIR executable.

---

Summary

This was a quick introduction to the Defender for Endpoint Live Response feature and troubleshooting capabilities. In one of the next blogs, I will deep-dive more into the automation part of Live Response including a couple of examples for collecting files via Logic Apps and Sentinel. Hopefully, this blog gives some ideas on how to troubleshoot Live Response failures.

---

Sources

Microsoft: Run live response commands on a device

Microsoft: Cancel machine action API