bluteam

Getting into the Blue Team: A Practical Guide

Anton Ovrutsky

• 11 min read

Are you a person who is new to the Information Security industry and want to get deeper into the defensive side of our wonderfully broad and complex industry? Have you read a few "getting into InfoSec" guides but been looking for something more practical, specific, and applicable to your interests in blue team aspects of InfoSec? If this sounds like you, then this blog post will be helpful. In it, we hope to cut through some noise and to provide our readers with helpful resources and insights on how to enter and progress in the blue team space.

Definitions

If you are new to the industry, you may qucikly become overwhelmed with the amount of career paths and various colors that get assigned to them. There’s a red team, blue team, white team, white hats, black hats; it’s all overwhelming. In the context of this post, a "blue team" can be defined as a career focus on the more defensive oriented aspects of information security. Some examples include detection engineering, digital forensics and incident response, security operations center (SOC) analyst and a plethora of other roles. A simplified way to think about these concepts is that someone who is on the "red" side of things would spend their time thinking about how to best attack systems in order to demonstrate weaknesses or misconfigurations and someone on the blue side of things would spend their time thinking about how to defend and harden such systems. Of course, this neat dichotomy exists on paper only, and as we will see, the lines between red and blue are often blurred.

Perspective

In InfoSec, critical thinking is a very important skill to exercise liberally. When reading and absorbing an article – particularly one such as this that offers career advice – it is always a good idea to critically interrogate the source of the information provided. With that in mind, this particular piece is authored by someone who has about a decade of InfoSec experience in various roles who has found a home on the blue team side of things. The aim here is not to provide definitive and authoritative guidance, but rather to make the sections of the pathway that I already travelled a little bit easier to navigate for those who are just starting this journey.

Skills

The answer to what skills are required in order to be a successful blue teamer will vary depending on the role and different organizational requirements, however, there are a few areas that we feel are worth highlighting.

Empathy

In your blue team travels you will undoubtedly come across networks with security gaps, processes that are not well defined, technology that is broken and non-functional. In other words, no environment is perfect and every environment is built and operated with a set of constraints. The ability to emphasize with people and organizations alike will help you identify these constraints and will help you build solutions for your clients or organizations. The ability to see things from the perspective of others is a critical skill to have as almost none of the work that you do as a blue teamer occurs in a vacuum. By understanding the challenges that are faced by your colleagues or clients, you will be in a better position to offer solutions and add value.

Humility

How many times have we all heard phrases like "the users are dumb" or "people in this particular business unit click on everything" – this is a mentality that needs to be tossed aside. A large part of being a successful blue teamer depends on the realization that a business constantly balances differing sets of priorities based on a limited set of resources and that security may not always be the top priority. Users are not stupid and are just trying to do their jobs, it is on us as defenders to ensure that a workable balance exists between security, productivity and user experience. Identifying gaps is only the first step in solving the security challenges faced by organizations, coming up with a remediation plan and working across various lines of business to execute this plan is what adds value.

Humility not only comes into play with the relationship between yourself and the organization, it also comes into play in interactions between colleagues and industry peers. A dynamic I have personally seen is those in technical InfoSec roles looking down on those in less-technical roles, such as governance, risk or compliance. Again, this mentality needs to be tossed into the garbage. This is an incredibly arrogant and simply inaccurate way of thinking. Those in less technical roles obviously serve important roles in organizations. The information security industry is home to some brilliant minds and it would be a wise decision to assume that everyone you interact with is more technical than you. In the same vein, no matter how much of a technical wizard you are, there will always be someone with more skill and knowledge. A little humility goes a long way indeed.

Curiosity

Many veteran InfoSec practitioners will highlight curiosity as a critical skill and this is applicable to blue teamers as well. What this means in practical terms is that a certain level of curiosity will help you in coming up with innovative solutions in the defensive space. Questions like: will this technique go undetected in my environment or what data source can I bring online to cover a detection gap are always prevalent in our industry and thinking about these questions with a curious mindset is an important first step in identifying solutions. Does this mean that you need to be curious about and tug on every thread of knowledge you have? Absolutely not, as that is a nice shortcut to burn out. However, tinkering with systems and data is a critical component to blue team efforts, many success stories in this space began with the phrase: "What if …."

The Technical Bits

A key thing to remember about technical skills is that you will never know everything about a certain operating system, product or technology – accepting this simple fact will help you maintain sanity throughout your career. As well, technical knowledge can be taught and looked up. That said, a certain level of foundational technical knowledge is required to be successful in this space. Keep in mind that this section has a strong "Microsoft Windows" tinge to it, as that is my particular area of expertise.

Networking

Networks are a foundation of modern enterprise environments and many job requirements will list networking knowledge as prerequisites for application. Often these requirements are vague and somewhat intimidating to new comers: Do I need to understand all of networking and be a routing ninja to become a successful blue teamer? The answer to that is no. What is required, however, is foundational and general knowledge regarding how the internet works and how networks are configured in modern enterprises. Consider the following example: you are employed at an organization whose network utilizes some kind of proxy for internet traffic – that is, when an endpoint wants to reach the internet, it first has to go through the proxy where the domain that the user wants to visit is checked for reputation and general maliciousness. The proxy has an IP of 192.168.1.5, you are reviewing logs from a host and you want to look at what websites the host visited, you look at network traffic logged by the EDR product installed on the host and only see requests to 192.168.1.5 which is the proxy IP. With basic networking knowledge, you would understand that if you wanted to see logs of websites visited, you would need to look at proxy and not host logs in this particular configuration.

Thinking of practical scenarios and working ‘backwards’ may help you focus your efforts in terms of learning and absorbing. The same idea can be applied to other aspects of the general networking stack as well, like Domain Name System (DNS) or Hypertext Transfer Protocols (HTTP) – there is nothing wrong with wanting to learn everything there is about DNS or HTTP, but we all have limited resources so thinking more along the lines of "How would I track a DNS request and response in my network" or "How do I interpret these Internet Information Services (IIS) logs" may break down these massive knowledge chunks into ingestible portions.

Operating Systems

Understanding how modern operating systems work is also a critical component of being a blue teamer. Does this mean that you need in-depth understanding of the Windows kernel to be a Windows-focused blue teamer? Nope. What is required knowledge, however, is understanding how attackers target Windows or Linux systems, what the common initial access vectors are and what popular techniques exist for exploiting these systems. In the Windows world, exploitation of some kind of software flaw is becoming less common and attackers turn to less hardened areas of the operating systems like malicious Office documents or various script interpreter payloads.

Understanding the telemetry that is available in these systems is also a critical skill. What kind of logging can be turned on a Windows host, what is noisy and what is valuable are great questions to be asking yourself. Speaking from experience, some of the best ways to learn is to tinker around in a lab environment.

Labs have been somewhat mythologized in InfoSec and if you have the desire and means to build a complex lab environment to learn in please do not let me discourage you. However, a simple lab consisting of a Windows Domain Controller and member host is more than sufficient to get started and would give you the opportunity to learn everything from Active Directory administration to basic threat hunting. You can install both Windows Server 2019 and Windows 10 as free inactivated trials to tinker around with.

Malware

Understanding different malware trends and how various malware families operate are critical skills to have as a blue team practitioner. Tons of information is published regarding malware – from a full reverse engineering writeup to actual full packet (PCAP) samples of trojans – all of this is freely available to us and serves as an invaluable source of information. Some questions to focus on when it comes to thinking about malware, from a blue team perspective: what kind of data do I need to detect these activities in my environment? Would this type of malware get around controls that exist in my organization or client site? Can I replicate certain portions of the way this malware behaves in a benign way in my test environment?

Tools

The blue team is fortunate in that we have a plethora of various tools available to us: EDR, XDR, SOAR, SIEM and literally hundreds of other acronyms get thrown around in the blue team space on a daily basis. Becoming an expert in every single SIEM or EDR is an impossible goal. What is more important to understand is how a particular tool fits into the overall organizational structure and what risk or problem it’s trying to address. If you have never looked at Windows logs before, the output that an EDR product will give you will undoubtedly make no sense to you. Likewise, tools come and go and are constantly evolving. Foundational knowledge is more important to have than knowledge regarding a specific toolset – for example, if you know what you are looking for, figuring out a particular query syntax for a SIEM product will be relatively straight forward for you. However, if you ignore the foundations and dive right into the toolset, you will find that you run into a dead end fairly quickly.

Cloud

The cloud computing space can be overwhelming. Google, Azure and Amazon all have major cloud offerings with hundreds, if not thousands, of services each. As a blue teamer the focus on cloud should be (non-exhaustively) around risks, identity protections and visibility. That is, what are the risks of a cloud implementation at your current organization or client. Similarly, identity is a critical component of cloud services so thinking of "cloud protections" in terms of identity may help focus efforts. Finally, telemetry and visibility are critical aspects required to secure cloud workloads. From a blue team point of view, the cloud can be overwhelming, but start small and start with bite-sized aspects like what do Azure sign in logs look like? How do I get these into my SIEM? What does a password spray attack look like in Azure? Can I see if someone provisioned a resource in AWS?

Offensive Security Tools

Many red team professionals publish their research and tooling publicly and blue teamers can take full advantage of this. Today, many Command & Control frameworks exist that can be freely installed and used. The ability to perform a semi-realistic attack in a safe manner within a lab environment that is configured to generate appropriate telemetry is the closest thing to a learning "fast lane" as we can think of. It is critically important that you understand what these tools do before you run them and you must exercise extreme caution and get appropriate approvals before running them at your organization or client site, but within a local and isolated lab environment, they can be an extremely powerful learning aid.

The Blue Team Candidate

We’ve gone over some technical and non-technical skill sets that aspiring blue teamers should be cognizant of. Putting this together, we start to see what a solid blue team candidate would look like – here are some things that we would look for in a blue team oriented role:

  • Understands that they are part of a larger organization working within a set of constraints
  • Offers solutions, rather than just identifying problems
  • Understands that they have more to learn, has no problem saying "I don’t know" and has the drive and willpower to identify and fill knowledge gaps
  • Aware of how networks are configured and has basic to intermediate practical networking knowledge
  • Knows what the risks to major operating systems are and their corresponding controls and telemetry sources
  • Able to identify risks to cloud workloads and their corresponding controls and telemetry sources
  • Is able to speak intelligently about the kinds of malware that are prevalent in networks
  • Is able to speak intelligently about how various tooling fits into the broader enterprise
  • Some familiarity with known offensive security tools like Metasploit or Cobalt Strike

Given the above set of skills, here are some sample interview questions that we would hypothetically ask of a blue team candidate:

  • A user has reported that they received a weird email with a document attached and when they opened the document nothing happened, what would you do?
  • You come into work and the SIEM is down, what are your next steps?
  • What log sources would you look at on a Windows host to see if a certain command executed?
  • Tell me about what offensive technique you find most interesting and why?
  • How would you go about investigating malicious PowerShell on a Windows host?

These questions are open ended on purpose and are not black and white, most do not have an absolute "correct" answer and the replies will obviously be gauged according to the targeted role and required seniority levels.

Contributing

Not every person is comfortable writing a blog post or giving talk in public and that is perfectly okay. However, contributing to the community can be an awesome way to not only raise your own personal capital, but also network with and meet people who may be able to help you along your journey.

If you enjoy blogging but have thought to yourself that you have nothing to contribute: please stop. Not every blog post needs to be an earth-shattering technical masterpiece, even a blog about something that you tried and could not get working or are struggling with would be valuable to many. It’s also worth remembering that you may see a polished blog post presented to you, but what you do not see is the mess of notes, failed attempts at getting the ‘thing’ to work properly and the weeks or months that it took to get all the materials together.

There are other ways to contribute as well, think about contributing a Sigma rule or sharing a quick detection rule that you wrote on Twitter.

The Grind

Making changes in organizations often takes a lot of time. Before implementing that shiny new product or kick ass control, paperwork needs to be signed and internal processes need to be worked through and it often feels like things are not moving in a forward direction. In these cases, it may be helpful to think of yourself as a drill operator that has been operating a giant earth-eating drill, digging a tunnel from point A to point B. You’ve been drilling for many months and all that you see in front of you is rock and earth. However, if you were to stop for a bit and turn around, you would see a long and empty tunnel and the fruits of your labor. The same concept can be applied to your daily work, sometimes in order to identify the impact you make, you need to turn around and take in all that you have accomplished.

On the Shoulders of Giants

It was mentioned earlier that the purpose of this blog post was to make the blue team path a bit easier to navigate for people who are on their way into this corner of InfoSec, however, it needs to be mentioned that this field is built on and benefits from the efforts of others who innovated in this space before I began my journey. This list of folks is not complete and is in no particular order, but these individuals all helped me directly or indirectly and continue to push the blue team forward:

If you are new to the InfoSec blue team space and are looking to build out your follow and watch list, I would highly recommend taking a look at these accounts. You may also want to sign up for our monthly newsletter as a way to stay informed on the recent tools, techniques, and practices related to the art of blue teaming.