Author: Rhys Thornton, Solutions Consultant (EMEA), Snare

Post-Breach Remediation 

Blocking a cyber threat before it has a chance to gain a foothold within your corporate infrastructure is the primary objective of many leading tools on the market today (including Snare), but what happens when those tools fail to detect a threat actor on the first pass? Whether it’s an entirely new threat, or a variation on an existing one, it is just as important to have a solid remediation process and tools to confirm your company data is not at risk. 

With the FBI and CISA both issuing a warning regarding an increase in the number of ransomware attacks (FBI and CISA Issue Conti Warning – Infosecurity Magazine (infosecurity-magazine.com)) being confident in your remediation process is vital. 

Snare products, combined with the enhanced logging capabilities of Sysmon, can provide enriched log data that can assist organisations with ensuring any remediation attempts have been successful. By monitoring the IOCs left behind by an existing breach, Snare and Sysmon can not only check for application hashes and malicious DNS requests across your infrastructure, they can also alert you to these events in realtime. By teaching Snare what IOCs to look for, realtime alerts can be triggered when any system in your organisation exhibits symptoms of the existing breach. 

In this article I am going to focus on 2 specific features from Sysmons logging capability, these are:  

Event ID 1: Process creation 

The process creation event provides extended information about a newly created process. The full command line provides context on the process execution. The ProcessGUID field is a unique value for this process across a domain to make event correlation easier. The hash is a full hash of the file with the algorithms in the HashType field.  

Event ID 22: DNSEvent (DNS query) 

This event is generated when a process executes a DNS query, whether the result is successful or fails, cached or not. The telemetry for this event was added for Windows 8.1 so it is not available on Windows 7 and earlier. 

More details on other events can be found here: Sysmon – Windows Sysinternals | Microsoft Docs 

Using these 2 events, we will be able to search our entire estate for any systems running a malicious process with a matching hash, as well as monitoring for any known malicious DNS requests that the application might make. This allows us to verify that remediation attempts have been successful (if we have no further events logged) or allows us to jump into action if we do see this occurring on other systems.   

Pre-requisites  

The following pre-requisites are required to follow along with the article:  

  • Sysmon needs to be installed on all systems and correctly configured to capture the required events. 
  • Snare agents need to be installed along side Sysmon. Along with the correct audit policies to capture Sysmon events (default Snare Agent audit policies will capture this)  
  • Snare Central installed and configured as a destination for our Snare agents. This enables us to monitor, search and get realtime alerts on logs as they are created. 

Disclaimers 

This article is designed to give an indication of the capabilities of Snare and Sysmon combined. The steps detailed in this article may not be valid for all cyber threats and each threat should be evaluated on a case-by-case basis with security professionals. 

Threat 

For the purposes of this article, I have created a simple program called “MaliciousApplication”. This application doesn’t do anything particularly special other than make specific DNS requests. As part of the initial breach, we identified this program to be the root cause of our issues and the DNS requests it makes are going to C&C servers on the internet. 

Steps have been taken to isolate and remediate the initial machine, but we don’t know if the malicious application had the opportunity to spread throughout our systems. Instead of requiring manual remediation/investigation across our entire estate (which can take considerable time) we can monitor for the particular IOCs (program hash and DNS requests) and check/monitor our entire estate for them. 

Searching estate for persistent threats 

As per the pre-requisites, I have Snare agents and Sysmon installed and configured on all my systems. All Snare agents are configured to send logging data to Snare Central, which is the centralised repository for event data.  

During the initial remediation we identified the below “Process Create” log as the initial inception point for the malicious application.   

Post Breach Remediation_1

Along with malicious DNS request below:

Post Breach Remediation_2

These 2 logs provide a huge amount of detail, along with a number of key data points we can use to monitor our infrastructure.  

Key data points to monitor:  

  • ParentCommandLine, this allows us to see the process that initiated the malicious code. Allowing us to further trace the root cause (for this article it will be Explorer.exe as I manually opened the program) 
  • Hashes, this provides us with a hash of the malicious program and a way to search for similar instances across our estate 
  • CommandLine, this allows to see any extended options specified in the startup of the application. This can sometimes contain extra information relevant to the breach. 
  • ProcessGuid, this allows to correlate other events to this process (such as DNS events) so we can perform a search for the GUID and get all events that relate to the process. 

Using the “Hashes” value stored within the payload of the log, we can utilise the search functionality of Snare Central to look for a process running with a matching hash.  

As you can see from the below screenshot, I have entered the hash into the search: 

Post Breach Remediation_3

As I have “All Systems” selected, Snare Central will look for this hash in any logs generated from all systems. If we receive any results, it indicates that the malicious application was able to replicate itself and we need to take remediation steps immediately. 

Th below results table shows there are 3 instances of this process being started on another system (the 4th log is the initial breach). 

Post Breach Remediation_4

To see any other actions this process took, we can expand out the record to see its details and select the ProcessGuid. 

Post Breach Remediation_5

Then perform another search on the ProcessGuid, the screenshot below shows the results:  

Post Breach Remediation_5

Here we can see the entire lifecycle of the process. When it starts, the DNS requests it makes to “malicious.int.rhyss.uk” and finally the process exiting. We also have the IP address of the system stored in the “System” field, making It easy to identify affected machines. 

Realtime alerts when IOCs occur 

Snare Central also has the capability to monitor inbound log data and trigger email alerts in the event a specific log is received. This is completely configurable using the report builder and can allow us to receive an email alert as soon as a process starts that matches the malicious hash. 

 To do this I am going to create a new report called “MaliciousApplication Alert”. Then configure the report as per the below screenshot: 

Post Breach Remediation_7

I have specified some criteria on the report to only match records created by Sysmon and its “Create Process” event ID. The hash is then passed in to ensure we only return results that match.  

Finally, I have added the “Realtime Alert” object to the report and ticked “Send Email Alerts”. This means as soon as an event is received that matches the above criteria, an email will be sent to any recipients I specify. 

Below is a screenshot of the email generated after running the malicious application on another system. 

Post Breach Remediation_8

Here I can see all the information related to this particular log, such as the date/time of the event and the system the event occurred on. This provides all the information needed to start remediating the affected system immediately. 

Minimise Negative Financial Impacts of System Downtime

With the number of cyber security incidents growing, it has never been more important to ensure that post breach, all remediation attempts have been effective. This ensures the business can regain trust within IT operations and minimise any negative financial impacts of system downtime. The steps outlined in this article can be applied to a number of common threats being actively exploited today, providing organisations with the detailed logging data they need to effectively remediate. 

The powerful combination of Snare and Sysmon provides organisations with detailed logging data they need to be effective at this.  

 

If you would like to know more, please use the contact our team, here.

Log collection and storage integrity is critical for both accurate real time threat detection and meeting compliance mandates (like PCI DSS, HIPAA, SOX, ISO27001, NIST, FISMA, NERC, etc), as well as current and evolving cyber industry standards for log storage and retention. Accurate forensic investigation cannot be done if critical event log data is missing or corrupt.

That is why Snare Central has introduced our new High Availability mode in Snare Central 8.4, enabling certainty of log collection even of your primary log collection and reporting server becomes unavailable. The Snare Collector/Reflector will automatically fail-over in this new HA mode, which means we are always collecting logs that are being forwarded for storage, analysis, and forensics.

Ensuring the integrity of log data is also critical if log data is needed as evidence in court – most likely in the event of a large scale breach. Being able to demonstrate integrity of collection means that if your log data is presented as evidence and is shown to have tamper protections, your organization will be less open to legal challenge.

The NIST Standard for log collection (NIST 800-92) states that:

“because logs contain records of system and network security, they need to be protected from breaches of their confidentiality and integrity”.

“Logs that are secured improperly in storage or in transit might also be susceptible to intentional and unintentional alteration and destruction. This could cause a variety of impacts, including allowing malicious activities to go unnoticed and manipulating evidence to conceal the identity of a malicious party. For example, many rootkits are specifically designed to alter logs to remove any evidence of the rootkits’ installation or execution.”

This is one of many reasons are why it is good security practice to store your logs on a system away from the system that generated the log data.

Snare has multiple layers of functionality to ensure log collection & storage integrity:

 

  • High Availability mode – the Snare Central Collector/Reflector can operate in a fail-over mode to ensure logs continue to be collected even if the primary reflector/collector service is interrupted from a hardware, network or system failure.
  • Mutual Authentication between Snare Agents and Snare Central/Reflector – using TLS Auth Agents and Snare Central are two communicating sides that exchange messages to acknowledge each other, verify each other, establish the encryption algorithms they will use and agree on session keys.

This ensures that logs are only collected from an authenticated and validated end point and are only sent to an authenticated and validated storage location.

  • Snare Agent Heartbeat – The agent can send out regular heartbeats, letting the collecting device know that the agent is working without having to make contact.  Agent logs are available, which allows the agent to send status messages to the collection device, such as memory usage, service start and stop messages, and any errors or warnings triggered during operations.  Snare Central has a health checker that can report on when agents are not sending logs or not. Often clients will use the Agent Heartbeat feature to ensure that logs are sent to the system to ensure you know it is online and logging.

This is of higher importance for end points that do not generate high numbers of logs and may only generate events sporadically.

  • Data EncryptionSnare is using the strongest publicly available encryption and hashing methods. It is using FIPS compliant cryptographic functions and TLS1.3 encryption. Certificate pinning and signing validation also helps to provide assurance that logs are encrypted in-transit to be kept private, to prevent tampering, and to ensure the encrypted session is a trusted path with no tampering or session highjacking.
  • Event Log Caching – When the Snare Windows Agent is unable to connect to the storage and collection server, it maintains a bookmark of the last sent Windows log event and waits. The events aren’t cached separately by the Agent, but rather it waits until the server is ready before continuing to read the log and send more events through.

This means no extra space is taken up by Snare specifically for log events, rather that the space is used by the Windows event log with the cache size increased as required for long periods where the Agent cannot talk to the server.

Other Snare Agents for other Operating Systems (Linux/Unix etc) create a separate cache as needed.

Snare Central reflector has both memory and disk caches for helping to manage the log data collection. If a system is a priority destination and there is a network or server disruption on the destination it will cache logs locally and use a disk cache during a prolonged outage and the memory cache is insufficient. Once the system comes back online, the Snare Reflector will send the events to the destination and drain the local cache. The disk cache usage is fully configurable by the customer, so they can create a cache as large as they need based on their available disk to cover expected outage scenarios.

This helps to provide assurance that the log data will be collected and forwarded as soon as possible from as many systems as possible with minimal chance of log data being lost.

Separation of Duties

  • Separation of Duties – configuration, security and log monitoring policy is controlled by the Agent and the Security Administrator and cannot be interfered with by the server admin. The Snare Agent logs can be stored on a Snare Central Server that has restricted access, and keeps the logs away from the system that generated them. Users don’t have access to these logs. Access is restricted to the defined admins who have console access. Access via the Agent UI does not allow any change access to the data on the server and can be restricted to privileged users. Once the logs are written to disk they never change and kept in a read only repository.  The logs are hashed to monitor for changes to the storage locations or operating system components.  The logs from the agents also have sequence numbers to make it easier to see if a log was missing and can also have additional hash details of each event embedded in the event to cover any potential of tampering with the content of the log before storage. All of these options help to provide assurance that the logs cannot be tampered with without some part of the integrity chain alerting to the fact.

“…having someone other than a system administrator review the logs for a particular system helps to provide accountability for the system administrator’s actions, including confirming that logging is enabled. NIST 800-92)”

The Snare Central server file and system checksum store capability can be downloaded to keep offline and burnt to DVD. This process helps to provide assurance that the logs have not been tampered with by an admin. These are automatically generated by the system health checker which can send email alerts for integrity failures. By keeping a copy of the hash details offline away from the system it can also help provide assurance and forensic details that the logs have not been tampered with and can be independently validated for forensic purposes if needed.

When the integrity of your security data is important (and its always important), Snare has you covered.

 

We’re Here to Help

We understand that security teams are under increasing pressure to enhance their organisation’s cyber security posture in the wake of many global cyber incidents. Contact your regional Snare office to ask how we can help or request a free trial to test Snare for yourself.

Snare XDR and Sysmon

Threat detection software has evolved significantly in recent years. Malware detection and prevention software began incorporating endpoint detection and response (EDR) in a change of tactics in response to more comprehensive, dangerous, and self-masking malware variants. The market has adopted a few different flavours of Detect/Response mechanisms – EDR, Network Detection and Response (NDR), Threat Detection and Response (TDR), and now the industry has settled more on Extended Detection and Response (XDR).

The terms have meant different things to different customers and vendors, but the overall goal is to detect and react to threats and unauthorised user activity across networks, servers and endpoints, clouds, and applications. XDR applies analytics and automation to detect, analyse, and hunt for threats in order to remediate or respond to identified threats.

By collecting forensic and log data from a diverse range of system components, correlated data can be scanned to identify patterns across event sources and provide context to a threat or attack chain. Events and activities that would not have been detected or addressed prior to XDR tools and concepts being implemented will be more obvious, allowing security teams to focus on, respond to, and mitigate/eliminate identified threat. This has the benefit of reducing further impact, and dropping the severity, scope, and longevity of the attack.

How Snare helps with XDR

The Snare suite has a comprehensive capability to collect log data from almost any source. Snare agents run on most common platforms such as Windows, Linux variations, MacOS, Solaris, Microsoft SQL Server, and can collect relevant forensic log activity from a wide range of systems and applications. The Snare agents collect and send log data to a Snare Central server and can send or reflect data to a range of other SIEM and analytics tools. The collected logs provide visibility into a range of security-sensitive activities, including:

  • Administrative Activity – What actions were undertaken by administrative users? Did they add/remove users, change permissions, change system policies, run privileged commands, etc.?
  • Login/Logoff Activity – Who logged on, was it within business hours or out of hours, what systems were impacted? Was it local or remote via a VPN or remote connection to the host? What was the source? Was there lateral user movement between systems? Are users trying to gain privileged access to systems? Are there indications of brute force logins occurring on the network to attempt to guess a password?
  • Command Activity – What applications or shell commands were run, were any executed at a higher privilege level? Where were these commands run, on what systems?
  • Data Access – What data was accessed, by who? What did they read, change, or delete? Was data copied or exfiltrated to other systems or out over the internet? Was a file or payload or backdoor loaded by malicious actors for later use?

There are various capabilities that help facilitate the analysis of these activities:

  • Collect logs from as many sources as possible – all servers, desktops, network devices, everything that can send a syslog. All devices should have some form of logging or monitoring in place.
  • Use FIM, FAM, RIM and RAM to track and monitor all key files and system configuration. Know who and when files were changing and what tools they used.
  • Use Database Activity Monitoring (DAM) to track key activity on SQL databases. Know if admin accounts are being abused and validate that key data has not been tampered with.
  • Having evidence to show if the attack vectors came in via email, USB, a web link download, a software update are all important to knowing how they got in.

To take a deeper dive into how Snare can help with XDR, download our technical white paper.

Background on the Print Spooler Vulnerability

The recent threat posted by Microsoft for a print spooler vulnerability is subject to exploitation from tools such as Mimikatz, a tool that can steal user credentials and potentially facilitates lateral movement of an attacker in the network.

PrintNightmare directly affects the native Windows service named “Print Spooler” that is enabled by default on Windows machines. The purpose of Print Spooler is to manage printers and printer servers.

  • SANS has published a recent article here.
  • CISA have also noted the issue here.

This covers the remote code exploitation of systems from the vulnerability. As of the 2nd of July, Microsoft has not fully patched the problem and still rates it as critical. Microsoft has published some workarounds to allow customers to potentially avoid the issue by reducing services/capabilities.

The process of exploiting the vulnerability will generate several print spooler events (event ID 316) in the PrintService-Operational custom windows event log.

How Snare Can Help

The Snare Enterprise Windows agents will collect any custom Windows event logs, that have been enabled by the administrator out of the box. The generation of PrintService custom logs can easily be enabled from either the Windows event viewer or from the group policy setting, at which time they will be picked up by the Snare agent and forwarded to your collection target system.

Open the Windows Event Viewer application then navigate to the Application and Service Logs → Microsoft → Windows → Then scroll down to PrintService and expand to see the Operational log then right click to enable.

An example as follows:

Mimikatz PrintNightmare_Example

Once enabled, logs will appear in this log and the Snare Agent will collect and send the logs to the Snare Central or your other target SIEM system.

In Snare Central, a specific report can be created for these events and a real-time alert can be triggered when one or more target events appear over a designated timeframe.

Create a new report using the menu using the “Add new objective” button in a report container of choice.

Change the log type to MS Windows Event Log – Adhoc query

An example of a report is provided below.

The source type has been configured with the name of the custom event log, the eventid has been set to 316 and the number of days set to 1 in order to limit the report to the most recent data. This can be adjusted to search for more days as needed:

Mimikatz Print Spooler Vulnerability

To display the events that meet these requirements, adjust the tabular reporting to show the following fields.

Mimikatz Print Spooler Field

Adjust the number of rows and pages to suit the number of rows being generated, keep in mind the larger the output the longer the report may take to generate.

For real time alerting based on a threshold of 2 events in 5 seconds you can set the criteria as follows:

Mimikatz Print Spooler Print Nightmare Snare Report

Select “Set” to save the report. Now set the schedule to provide the email address for the alerts to go to.

This will use the SMTP gateway as set in the configuration Wizard.

PrintNightmare Snare Detection

Once active, if any “316” eventIDs in the PrintService event log are received, a real-time alert will be generated.

 

Need help from Snare’s Team?

Contact our team if you have questions about the PrintNightmare incident or how to set up Snare to manage or audit printer server logs.