penetrationtesting Featured

Introducing Scraping Kit

Scraping Kit comprises several tools for scraping services for keywords, useful for the initial enumeration of Domain Controllers or if you have popped a user's desktop with access to their Outlook client.

Neil Lines, Andy Gill

• 14 min read
Introducing Scraping Kit

The Scraping Kit comprises several tools for scraping services for keywords, useful for the initial enumeration of Domain Controllers or if you have popped a user's desktop with access to their Outlook client. This project is a work in progress, and other functions will be added over time.  

GitHub - LaresLLC/ScrapingKit: Scraping Kit is made up of several tools for scraping services for keywords, useful for initial enumeration of Domain Controllers or if you have popped a user’s desktop, their outlook client.
Scraping Kit is made up of several tools for scraping services for keywords, useful for initial enumeration of Domain Controllers or if you have popped a user's desktop, their outlook client. -…
GitHubLaresLLC

Tell Me More

Back story, the Outlook application has always given off an interesting vibe; it’s just one of those apps which feels like it could be used for fun.

This post reveals some of our recent research into an interesting approach around leveraging Outlook COM Objects to interact with the Outlook desktop application.

For your convenience, we have supplied examples of our research in both PowerShell and C#.

The following lab was used for this project:

Domain - hacklab.local
Domain Controller - Microsoft Windows Server 2022 10.0.20348 N/A Build 20348
Domain Host 1 - Microsoft Windows 11 Enterprise 10.0.22621 N/A Build 22621
Domain Host 2 - Microsoft Windows 11 Enterprise 10.0.22621 N/A Build 2262

Microsoft Outlook for Microsoft 365 MSO (Version 2304 Build 16.0.16327.20200) 64-bit
Microsoft Outlook 2019 MSO (Version 2304 Build 16.0.16327.20200) 32-bit
Host Defender settings

Show Us Your Credentials

The ‘ScrapeKit’ C# console application so far offers users the following three options.

Active options

Selecting option 1 Run Outlook Email Search results in a response requesting a destination email address, any matches to your keywords will be forwarded to this user-defined address.

Entering an email address followed by pressing enter

Following this, the user is requested to answer if they would like to select the default keywords, which consist of defaultKeywords = { "password", "security", "confidential", "VPN", "WIFI" }.

Default Keywords Option

A response of Y instructs the code to start scraping the default keywords across all emails stored within folders across the user's Outlook application.

Matching emails are starting to be forwarded

An example of an email received was sent because of the matched keyword of password.

Received Email body containing matching keyword

Going back to the menu options, if a response of N rejecting the default keywords was selected, you are then allowed to add your keywords for scrapping; in this example, the following keywords were inputted cat, dance and test.

User defined keywords

The email was forwarded because the user-defined keyword of cat was matched in the body of the original email.

But There’s More

Domain Controllers (DCs) offer various opportunities for exploitation. By default, any user who is part of the domain users' group and has access to a domain-joined host can establish a share on a domain controller. Through this connection, they can effectively enumerate the present directories, which typically consist of NETLOGON and SYSVOL as the default shares.

During the initial phase of a test, it is recommended to employ a manual approach for mounting and enumerating all accessible shares on a DC. This can be accomplished by browsing to the Fully Qualified Domain Name (FQDN) of the domain in Windows Explorer \\hacklab.local\.

Access to the shares on a DC

The NETLOGON share on a Domain Controller (DC) is important in a Windows domain. Its primary purpose is to provide a centralized location for storing and distributing logon scripts to domain computers during user authentication. Logon scripts are typically written in scripting languages such as batch scripts (e.g., .bat files) or PowerShell scripts (e.g., .ps1 files). These scripts can automate tasks, such as mapping network drives, configuring user settings, or executing specific actions upon user logon.

Storing sensitive credentials in scripts within the NETLOGON share is not an uncommon practice. However, it poses a significant security concern due to the accessibility of the share by authenticated users within the domain. This means anyone with access to the share could view the scripts' contents, including any embedded credentials.

Example of hardcoded credentials in a script

Automation

We cannot strongly enough recommend that any exposed share should be manually reviewed for any documents or scripts. The process is trivial and enjoyable and yields significant insights into the domain's infrastructure. Moreover, this practice often leads to the capture of privileged credentials, but there is always a but; you sometimes don’t have time, or you fancy an automated solution to check you haven’t missed anything.

There are loads of scripts that will enumerate shares, and some even offer credential-scraping services. We love Snaffler here at Lares  https://github.com/SnaffCon/Snaffler, which is an impressive tool, but it got me thinking, could I write my own simpler solution in C# or PowerShell?

Enter the second part of ScrapeKit. It also includes an option titled Run Active Directory Keyword Search if this option is selected, the user is prompted to add at least one keyword.

Option 2 Run Active Directory Keyword Search

After adding keywords, the application enumerates the local domain controller (DC), connects to it, and performs a scrape of the chosen keywords against all files available across the SYSVOL and NETLOGON directories.

The following example shows the result of adding golf and cpass as keywords for the DC scrape.

The keyword of cpass was matched

A match of cpass (GPP vulnerability) was found in the file \\WIN-4Q0A4190APL.hacklab.local\SYSVOL\HACKLAB.LOCAL\Policies\{31B2F340-016D-11D2-945F-00C04FB984F9}\MACHINE\Test\Groups.xml!

And the matching lines are also presented on the console for ease of use.

The keyword of golf was matched in the file \\WIN-4Q0A4190APL.hacklab.local\SYSVOL\HACKLAB.LOCAL\Scripts\Script99.txt!.

The keyword of golf was matched

Quick Overview of the Outlook email search functionality:

  • The user is prompted to enter an email address to which matching emails will be forwarded.
  • Users are requested to select whether they want to include the default keywords of 'password', 'security', 'confidential', 'VPN', and 'WIFI' or enter their keywords for the search.
  • The console application communicates to the desktop Outlook application using Component Object Model (COM) interop, which enables interaction with Outlook features.
  • The inbox folder is enumerated and recursively processes each email from the inbox and any subfolders.
  • Any matching keywords result in the email being forwarded to the specified forwarding email address. The subject and body of the forwarded email include the original email's information, such as the sender, recipients, attachments, and the email body itself.
  • A console message indicates that a matching email was found and forwarded.
  • Resource leaks are avoided by releasing the COM objects.
  • Clean up process deletes all created, Sent and Deleted Items.
  • Returns to the original menu.

The Code

https://github.com/LaresLLC/ScrapingKit/tree/main/SharpScrapeKit

C# Errors and Troubleshooting

When you attempt to compile the code, you may encounter the following error:

“CS0234 The type or namespace name 'Office' does not exist in the namespace 'Microsoft'”.

This is caused when the required assembly or reference for the Microsoft Office object model is not properly included in the project. To fix this, add the Microsoft.Office.Interop.Outlook Namespace from the .NET Framework that provides the classes and interfaces required to interact with Microsoft Outlook.

Right-click on project 'ScrapeKit' and select Manage NuGet Packages
Click on Browse and search for Microsoft.Office.Interop.Outlook

The additional packages were also added to the application.

The following options were selected while publishing

Still have a love For  PS

Different circumstances require different approaches, so have included the original PowerShell one-liner I created at the start of this project in both Outlook and DC flavours.

https://github.com/LaresLLC/ScrapingKit/blob/main/PSScrapeKit/

What Does the PS Script Do?

The script scans files in the SYSVOL folder for potential passwords and usernames. It analyzes the content, identifies matches, extracts relevant information, and presents the findings.

For a more detailed overview of each section of the script please read the below.

The $domain = $env:USERDNSDOMAIN; line sets the $domain variable to the current user's DNS domain.

The $domainController = ([System.DirectoryServices.ActiveDirectory.Domain]::GetCurrentDomain()).DomainControllers | Select-Object -First 1 ; line enumerates the current domain controller and adds the name to the $domainController variable.

The $netlogonPath = "\\$($domainController.Name)\SYSVOL\$domain"; sets the $netlogonPath variable to the path of the SYSVOL folder on the domain controller.

This line $initialKeywords=@('password','cpassword','passw','cred','Password','Cpassword','Passw','Cred','Password:','password:','Password=','password=' ,'password ','cpassword ','passw ','cred ','Password ','Cpassword ','Passw ','Cred ','Password: ','password: ','Password= ','password= ','Password : ','password : ','Password = ','password = '); creates an array of strings for potential keywords for passwords.

This line $additionalKeywords=@('user','username','name','User','Username','Name','Username:','username:','Username=','username=','user ','username ','name ','User ','Username ','Name ','Username: ','username: ','Username= ','username= ' ,'Username : ','username : ','Username = ','username = '); does the same, but the array of strings are for potential usernames.

The $matchesFound=$false; line sets the $matchesFound variable to false for matches that haven't been found yet.

The Get-ChildItem -Path $netlogonPath -Recurse -File | Where-Object { $_.Name -notin @('GptTmpl.inf','GPT.INI','Registry.pol') } | ForEach-Object { line gets all the files in the $netlogonPath directory and its subdirectories, excluding the following file names 'GptTmpl.inf','GPT.INI','Registry.pol' because they don’t contain credentials but reference the word password. It then loops through each file.

The $content=Get-Content $_.FullName; the line reads the contents of the current file into the $content variable.

The foreach($line in $content){ line loops through each line in the file.

The $matches=$initialKeywords|Where-Object{ $line -cmatch $_ }; line checks if the current line contains any initial keywords for passwords. If it does, it sets the $matches variable to true.

The if($matches){ line checks if any matches were found.

The $matchesFound=$true; line sets $matchesFound to true, indicating that at least one match was found.

The Write-Host "Match found in file $($_.FullName)!"; the line displays a message indicating that a match was found in the current file.

The $contextStart=[Math]::Max(0,[Array]::IndexOf($content,$line)-3);$contextEnd=[Math]::Min([Array]::IndexOf($content,$line)+3,$content.Count-1);$context=$content[$contextStart..$contextEnd]; line creates a context variable that contains the three lines before and after the matching line.

The $additionalKeywordsFound=$additionalKeywords|Where-Object{ $context -like "*$_*" }; line checks if any additional keywords for usernames were found in the context.

Execution

The script response can be quite extensive, so it is recommended to use an output redirection by adding the > character followed by any file name you wish, for example > test23.txt this results in any output that would typically be displayed via the console being redirected and written to the specified file instead.

Using an output redirection on the end of the script records the response to a file
PS C:\Users\user1> $domain = $env:USERDNSDOMAIN; $domainController = ([System.DirectoryServices.ActiveDirectory.Domain]::GetCurrentDomain()).DomainControllers | Select-Object -First 1 ; $netlogonPath = "\\$($domainController.Name)\SYSVOL\$domain";$initialKeywords=@('password','cpassword','passw','cred','Password','Cpassword','Passw','Cred','Password:','password:','Password=','password=' ,'password ','cpassword ','passw ','cred ','Password ','Cpassword ','Passw ','Cred ','Password: ','password: ','Password= ','password= ','Password : ','password : ','Password = ','password = ');$additionalKeywords=@('user','username','name','User','Username','Name','Username:','username:','Username=','username=','user ','username ','name ','User ','Username ','Name ','Username: ','username: ','Username= ','username= ' ,'Username : ','username : ','Username = ','username = ');$matchesFound=$false;Get-ChildItem -Path $netlogonPath -Recurse -File | Where-Object { $_.Name -notin @('GptTmpl.inf','GPT.INI','Registry.pol') } | ForEach-Object { $content=Get-Content $_.FullName;foreach($line in $content){$matches=$initialKeywords|Where-Object{ $line -cmatch $_ };if($matches){$matchesFound=$true;Write-Host "Match found in file $($_.FullName)!";$contextStart=[Math]::Max(0,[Array]::IndexOf($content,$line)-3);$contextEnd=[Math]::Min([Array]::IndexOf($content,$line)+3,$content.Count-1);$context=$content[$contextStart..$contextEnd];$additionalKeywordsFound=$additionalKeywords|Where-Object{ $context -like "*$_*" };$username=$line|Select-String -Pattern '(?i)username\s*[:=]\s*(.+)' -AllMatches|ForEach-Object{ $_.Matches.Groups[1].Value };if([string]::IsNullOrEmpty($username)){$username=$context -join ' '};$password=$line|Select-String -Pattern '(?i)(?:password|passw|cred)\s*[=:]\s*(\S+)' -AllMatches|ForEach-Object{ $_.Matches.Groups[1].Value };if([string]::IsNullOrEmpty($password)){$password=$content|Select-String -Pattern '(?i)(?:password|passw|cred)\s*[=:]\s*(\S+)' -AllMatches|ForEach-Object{ $_.Matches.Groups[1].Value }};if([string]::IsNullOrEmpty($password)){$password=$line};[PSCustomObject]@{FileName=$_.Name;FullName=$_.FullName;PrecedingContext=$context[0..($context.IndexOf($line)-1)];MatchingLine=$line;TrailingContext=$context[($context.IndexOf($line)+1)..($context.Count-1)];AdditionalKeywordsFound=$additionalKeywordsFound;Username=$username;Password=$password}}};if(-not $matchesFound){Write-Host "No matches found.";}} > test23.txt

Match found in file \\WIN-4Q0A4190APL.hacklab.local\SYSVOL\HACKLAB.LOCAL\scripts\Game2.txt!

Match found in file \\WIN-4Q0A4190APL.hacklab.local\SYSVOL\HACKLAB.LOCAL\scripts\Script99.txt!

Match found in file \\WIN-4Q0A4190APL.hacklab.local\SYSVOL\HACKLAB.LOCAL\scripts\Brandon_DiCamillo\Startup.bat!

Match found in file \\WIN-4Q0A4190APL.hacklab.local\SYSVOL\HACKLAB.LOCAL\scripts\Deep\In\The\Cave\Script_remove1.txt!

Match found in file \\WIN-4Q0A4190APL.hacklab.local\SYSVOL\HACKLAB.LOCAL\scripts\Test\Here_it_is\Readme.txt!

Match found in file \\WIN-4Q0A4190APL.hacklab.local\SYSVOL\HACKLAB.LOCAL\scripts\Test2\Dog_Cat\Script1.txt!

PS C:\Users\user1>

Then to review the results, open the file the script created.

Username and Password are harvested from file Startup.bat
A few examples of credentials identified across different scripts

Group Policy Preferences (GPP)

OK, we get it GPP is old, but in the real world, you can still occasionally encounter stored password hashes in the Groups.xml files located at \\hacklab.local\SYSVOL\hacklab.local\Policies.

For those new to this, GPP vulnerability exposes passwords stored within Group Policy scripts, making them easily accessible and decryptable by malicious actors.

Any domain users' group member can navigate to the SYSVOL Policies Directory on a domain controller.

Base location of the GPP files

Using Windows Explorer, initiate a search for any Groups.XML files.

Result of searching for Groups.xml

An example of an encrypted hash within the cpassword parameter contained within a legacy Groups.xml file.

cpassword parameter contains a password hash

The extracted hashes can be decrypted using https://github.com/t0thkr1s/gpp-decrypt, as seen in the following example.

ubuntu@NL-Work1:~/gpp-decrypt$ python3 gpp-decrypt.py -c j1Uyj3Vx8TY9LtLZil2uAuZkFQA/4latT76ZwgdHdhw

[ * ] Password: Local*P4ssword!

But There’s More

The PowerShell script created for this post not only enumerates all files and extracts potential usernames and passwords but also attempts to locate and extract usernames and hashes within any identified Groups.xml files.

The executed script reveals a captured GPP password hash

Outlook PS Execution

Copy and paste the PS into a session, then select from the options; in this example, option one was selected, allowing for user-defined keywords to be added, followed by a destination email address.

Keywords matched in email body and forwarded to the destination email address

A Program Is Trying To Send An Email Message On Your Behalf.

During this project, code execution attempts would occasionally be hindered by a warning message from the o365 Outlook client, requiring a manual confirmation by clicking the "OK" button before any code would execute.

Sounds good, right?

Initially, it was presumed that the alert came from PowerShell (PS), not being trusted by the Outlook Messaging Application Programming Interface (MAPI).

To try and overcome this intermittent restriction, the original PS script was converted into a Visual Basic Application Macro. However, on execution, the warning messages were still occasionally triggered.

C# was then considered a potential way to bypass the intermittent alert. I admit I have had no real-world experience with C#, but discovered that the transition from PS to C# was not as daunting as I had initially anticipated.

Anyway, back to the intermittent alert, and please hold my beer while I explain this win. So, it was discovered from reading more into the alert that it only triggers when the host's anti-virus is out-of-date or can’t communicate with the Outlook client.

This concluded that the intermittent nature of the alert was due to the habit of rebooting the laptop between test attempts, which restricted Defender from establishing communication with Outlook before execution.

For the record, the lab defender was updated daily throughout the project.

Defensive Advice

Educate users on the dangers of sharing authentication credentials via insecure channels such as emails. Emails are not a secure method for transporting or storing credentials; consider implementing a data retention policy for the initial setup of accounts and have the emails automatically deleted after a certain period. One of the key issues we find on Insider Threat and Red Team engagements, again and again, is poor data governance, as highlighted recently in Lares' insider threat findings post.

If PowerShell is not required for everyday work functions, it can be disabled. The following method can disable PowerShell versions 3.0 and above on a host machine.

Open the Start menu, search for "Windows PowerShell," and right-click on "Windows PowerShell" or "Windows PowerShell ISE." and choose "Run as administrator."

No, the irony of using PowerShell to disable itself is not lost on me.

To disable PowerShell for all users, paste in the following command and press Enter:

  • Disable-WindowsOptionalFeature -Online -FeatureName MicrosoftWindowsPowerShellV2Root

To disable PowerShell for the current user, type or paste the following command and press Enter:

  • Set-ExecutionPolicy -ExecutionPolicy Restricted -Scope CurrentUser

Wrapping Up

Hardcoding credentials into scripts and files pose significant risks, and it is strongly advised to avoid this practice due to the following reasons:

Hardcoded credentials are easily accessible to anyone with access to the script or file. This includes potential adversaries who may exploit the credentials to gain unauthorized access to sensitive systems or data.

Where possible, restrict access to scripts and files containing sensitive data to only authorized individuals who require access, and regularly review and update access permissions.

PowerShell provides the Get-Credential cmdlet prompts the user to enter credentials and securely stores them in the Windows Credential Manager. You can later retrieve these credentials within your script.

With .bat files, prompt the user to enter the credentials when the batch file is executed. You can use the SET /P command to receive input from the user.

Further Reading

While explaining this project to one of my colleagues, he said, “I wrote something similar to this”, And yes, he had. Please see the links here; ScrapingKit builds on both and adds more functionality; enjoy!