Author: Rhys Thornton, Solutions Consultant (EMEA), Snare
Post-Breach Remediation
Blocking a cyber threat before it has a chance to gain a foothold within your corporate infrastructure is the primary objective of many leading tools on the market today (including Snare), but what happens when those tools fail to detect a threat actor on the first pass? Whether it’s an entirely new threat, or a variation on an existing one, it is just as important to have a solid remediation process and tools to confirm your company data is not at risk.
With the FBI and CISA both issuing a warning regarding an increase in the number of ransomware attacks (FBI and CISA Issue Conti Warning – Infosecurity Magazine (infosecurity-magazine.com)) being confident in your remediation process is vital.
Snare products, combined with the enhanced logging capabilities of Sysmon, can provide enriched log data that can assist organisations with ensuring any remediation attempts have been successful. By monitoring the IOCs left behind by an existing breach, Snare and Sysmon can not only check for application hashes and malicious DNS requests across your infrastructure, they can also alert you to these events in realtime. By teaching Snare what IOCs to look for, realtime alerts can be triggered when any system in your organisation exhibits symptoms of the existing breach.
In this article I am going to focus on 2 specific features from Sysmons logging capability, these are:
Event ID 1: Process creation
The process creation event provides extended information about a newly created process. The full command line provides context on the process execution. The ProcessGUID field is a unique value for this process across a domain to make event correlation easier. The hash is a full hash of the file with the algorithms in the HashType field.
Event ID 22: DNSEvent (DNS query)
This event is generated when a process executes a DNS query, whether the result is successful or fails, cached or not. The telemetry for this event was added for Windows 8.1 so it is not available on Windows 7 and earlier.
More details on other events can be found here: Sysmon – Windows Sysinternals | Microsoft Docs
Using these 2 events, we will be able to search our entire estate for any systems running a malicious process with a matching hash, as well as monitoring for any known malicious DNS requests that the application might make. This allows us to verify that remediation attempts have been successful (if we have no further events logged) or allows us to jump into action if we do see this occurring on other systems.
Pre-requisites
The following pre-requisites are required to follow along with the article:
- Sysmon needs to be installed on all systems and correctly configured to capture the required events.
- Snare agents need to be installed along side Sysmon. Along with the correct audit policies to capture Sysmon events (default Snare Agent audit policies will capture this)
- Snare Central installed and configured as a destination for our Snare agents. This enables us to monitor, search and get realtime alerts on logs as they are created.
Disclaimers
This article is designed to give an indication of the capabilities of Snare and Sysmon combined. The steps detailed in this article may not be valid for all cyber threats and each threat should be evaluated on a case-by-case basis with security professionals.
Threat
For the purposes of this article, I have created a simple program called “MaliciousApplication”. This application doesn’t do anything particularly special other than make specific DNS requests. As part of the initial breach, we identified this program to be the root cause of our issues and the DNS requests it makes are going to C&C servers on the internet.
Steps have been taken to isolate and remediate the initial machine, but we don’t know if the malicious application had the opportunity to spread throughout our systems. Instead of requiring manual remediation/investigation across our entire estate (which can take considerable time) we can monitor for the particular IOCs (program hash and DNS requests) and check/monitor our entire estate for them.
Searching estate for persistent threats
As per the pre-requisites, I have Snare agents and Sysmon installed and configured on all my systems. All Snare agents are configured to send logging data to Snare Central, which is the centralised repository for event data.
During the initial remediation we identified the below “Process Create” log as the initial inception point for the malicious application.
Along with malicious DNS request below:
These 2 logs provide a huge amount of detail, along with a number of key data points we can use to monitor our infrastructure.
Key data points to monitor:
- ParentCommandLine, this allows us to see the process that initiated the malicious code. Allowing us to further trace the root cause (for this article it will be Explorer.exe as I manually opened the program)
- Hashes, this provides us with a hash of the malicious program and a way to search for similar instances across our estate
- CommandLine, this allows to see any extended options specified in the startup of the application. This can sometimes contain extra information relevant to the breach.
- ProcessGuid, this allows to correlate other events to this process (such as DNS events) so we can perform a search for the GUID and get all events that relate to the process.
Using the “Hashes” value stored within the payload of the log, we can utilise the search functionality of Snare Central to look for a process running with a matching hash.
As you can see from the below screenshot, I have entered the hash into the search:
As I have “All Systems” selected, Snare Central will look for this hash in any logs generated from all systems. If we receive any results, it indicates that the malicious application was able to replicate itself and we need to take remediation steps immediately.
Th below results table shows there are 3 instances of this process being started on another system (the 4th log is the initial breach).
To see any other actions this process took, we can expand out the record to see its details and select the ProcessGuid.
Then perform another search on the ProcessGuid, the screenshot below shows the results:
Here we can see the entire lifecycle of the process. When it starts, the DNS requests it makes to “malicious.int.rhyss.uk” and finally the process exiting. We also have the IP address of the system stored in the “System” field, making It easy to identify affected machines.
Realtime alerts when IOCs occur
Snare Central also has the capability to monitor inbound log data and trigger email alerts in the event a specific log is received. This is completely configurable using the report builder and can allow us to receive an email alert as soon as a process starts that matches the malicious hash.
To do this I am going to create a new report called “MaliciousApplication Alert”. Then configure the report as per the below screenshot:
I have specified some criteria on the report to only match records created by Sysmon and its “Create Process” event ID. The hash is then passed in to ensure we only return results that match.
Finally, I have added the “Realtime Alert” object to the report and ticked “Send Email Alerts”. This means as soon as an event is received that matches the above criteria, an email will be sent to any recipients I specify.
Below is a screenshot of the email generated after running the malicious application on another system.
Here I can see all the information related to this particular log, such as the date/time of the event and the system the event occurred on. This provides all the information needed to start remediating the affected system immediately.
Minimise Negative Financial Impacts of System Downtime
With the number of cyber security incidents growing, it has never been more important to ensure that post breach, all remediation attempts have been effective. This ensures the business can regain trust within IT operations and minimise any negative financial impacts of system downtime. The steps outlined in this article can be applied to a number of common threats being actively exploited today, providing organisations with the detailed logging data they need to effectively remediate.
The powerful combination of Snare and Sysmon provides organisations with detailed logging data they need to be effective at this.
If you would like to know more, please use the contact our team, here.
Log collection and storage integrity is critical for both accurate real time threat detection and meeting compliance mandates (like PCI DSS, HIPAA, SOX, ISO27001, NIST, FISMA, NERC, etc), as well as current and evolving cyber industry standards for log storage and retention. Accurate forensic investigation cannot be done if critical event log data is missing or corrupt.
That is why Snare Central has introduced our new High Availability mode in Snare Central 8.4, enabling certainty of log collection even of your primary log collection and reporting server becomes unavailable. The Snare Collector/Reflector will automatically fail-over in this new HA mode, which means we are always collecting logs that are being forwarded for storage, analysis, and forensics.
Ensuring the integrity of log data is also critical if log data is needed as evidence in court – most likely in the event of a large scale breach. Being able to demonstrate integrity of collection means that if your log data is presented as evidence and is shown to have tamper protections, your organization will be less open to legal challenge.
The NIST Standard for log collection (NIST 800-92) states that:
“because logs contain records of system and network security, they need to be protected from breaches of their confidentiality and integrity”.
“Logs that are secured improperly in storage or in transit might also be susceptible to intentional and unintentional alteration and destruction. This could cause a variety of impacts, including allowing malicious activities to go unnoticed and manipulating evidence to conceal the identity of a malicious party. For example, many rootkits are specifically designed to alter logs to remove any evidence of the rootkits’ installation or execution.”
This is one of many reasons are why it is good security practice to store your logs on a system away from the system that generated the log data.
Snare has multiple layers of functionality to ensure log collection & storage integrity:
- High Availability mode – the Snare Central Collector/Reflector can operate in a fail-over mode to ensure logs continue to be collected even if the primary reflector/collector service is interrupted from a hardware, network or system failure.
- Mutual Authentication between Snare Agents and Snare Central/Reflector – using TLS Auth Agents and Snare Central are two communicating sides that exchange messages to acknowledge each other, verify each other, establish the encryption algorithms they will use and agree on session keys.
This ensures that logs are only collected from an authenticated and validated end point and are only sent to an authenticated and validated storage location.
- Snare Agent Heartbeat – The agent can send out regular heartbeats, letting the collecting device know that the agent is working without having to make contact. Agent logs are available, which allows the agent to send status messages to the collection device, such as memory usage, service start and stop messages, and any errors or warnings triggered during operations. Snare Central has a health checker that can report on when agents are not sending logs or not. Often clients will use the Agent Heartbeat feature to ensure that logs are sent to the system to ensure you know it is online and logging.
This is of higher importance for end points that do not generate high numbers of logs and may only generate events sporadically.
- Data Encryption – Snare is using the strongest publicly available encryption and hashing methods. It is using FIPS compliant cryptographic functions and TLS1.3 encryption. Certificate pinning and signing validation also helps to provide assurance that logs are encrypted in-transit to be kept private, to prevent tampering, and to ensure the encrypted session is a trusted path with no tampering or session highjacking.
- Event Log Caching – When the Snare Windows Agent is unable to connect to the storage and collection server, it maintains a bookmark of the last sent Windows log event and waits. The events aren’t cached separately by the Agent, but rather it waits until the server is ready before continuing to read the log and send more events through.
This means no extra space is taken up by Snare specifically for log events, rather that the space is used by the Windows event log with the cache size increased as required for long periods where the Agent cannot talk to the server.
Other Snare Agents for other Operating Systems (Linux/Unix etc) create a separate cache as needed.
Snare Central reflector has both memory and disk caches for helping to manage the log data collection. If a system is a priority destination and there is a network or server disruption on the destination it will cache logs locally and use a disk cache during a prolonged outage and the memory cache is insufficient. Once the system comes back online, the Snare Reflector will send the events to the destination and drain the local cache. The disk cache usage is fully configurable by the customer, so they can create a cache as large as they need based on their available disk to cover expected outage scenarios.
This helps to provide assurance that the log data will be collected and forwarded as soon as possible from as many systems as possible with minimal chance of log data being lost.
Separation of Duties
- Separation of Duties – configuration, security and log monitoring policy is controlled by the Agent and the Security Administrator and cannot be interfered with by the server admin. The Snare Agent logs can be stored on a Snare Central Server that has restricted access, and keeps the logs away from the system that generated them. Users don’t have access to these logs. Access is restricted to the defined admins who have console access. Access via the Agent UI does not allow any change access to the data on the server and can be restricted to privileged users. Once the logs are written to disk they never change and kept in a read only repository. The logs are hashed to monitor for changes to the storage locations or operating system components. The logs from the agents also have sequence numbers to make it easier to see if a log was missing and can also have additional hash details of each event embedded in the event to cover any potential of tampering with the content of the log before storage. All of these options help to provide assurance that the logs cannot be tampered with without some part of the integrity chain alerting to the fact.
“…having someone other than a system administrator review the logs for a particular system helps to provide accountability for the system administrator’s actions, including confirming that logging is enabled. NIST 800-92)”
The Snare Central server file and system checksum store capability can be downloaded to keep offline and burnt to DVD. This process helps to provide assurance that the logs have not been tampered with by an admin. These are automatically generated by the system health checker which can send email alerts for integrity failures. By keeping a copy of the hash details offline away from the system it can also help provide assurance and forensic details that the logs have not been tampered with and can be independently validated for forensic purposes if needed.
When the integrity of your security data is important (and its always important), Snare has you covered.
We’re Here to Help
We understand that security teams are under increasing pressure to enhance their organisation’s cyber security posture in the wake of many global cyber incidents. Contact your regional Snare office to ask how we can help or request a free trial to test Snare for yourself.
