Considering an XDR Purchase? Here Are Our Lessons Learned.

Lessons learned from our search for, and integration of, our XDR

Trusted Internet is now deploying Stellar Cyber XDR –as a SOC-monitored solution or as an Infrastructure as a Service.

The marketing hype around XDR is deafening for those of you considering an XDR. It’s hard to sort through the slick websites and marketing noise to tell what’s actually real. So, I thought I share a few lessons learned –from the viewpoint of the CEO of a self-funded MSSP, I hope this helps in your buying decisions.

For the last four years, we’ve been a died-in-the-wool Fortinet MSSP. We love our Fortinet firewalls, with our people certified through NSE7, working hard to tune the feature-packed high-speed machines to bend to our will. For various reasons, we decided about two years ago to begin the search for a way to accommodate the requests from would-be clients to not have to rip and replace their existing security systems.

As well, SOC, NOC, EDR, MDR, NDR, MSSP. Why would someone not combine them all into one box that understands ALL of their logs and uses a bit of machine learning to train AI to better assist SOC analysts? I have an old friend that used to refer to this as the God Box. It knows all.

XDR is the beginning of the God Box.

Our requirements:

• ***It must integrate all those other vendors in a client’s environment without requiring them to rip and replace their existing infrastructure.***We didn’t want to have an agent deployed to every computer. They already have AV and Anti-Evasion. We didn’t want to load on another endpoint system.
  We want the ability to integrate network flow analysis for anomaly detection but may not want it 100% of the time. Flow produces heavy volumes of data that we wanted to be able to turn on and off as needed based on other indicators.

• ***It must accommodate all NIST 800-171 log-collection/analysis requirements.***While ISO, CIS, HIPAA, or PCI require the aggregation and analysis of all of these logs, NIST 800-171 requires monitored log entries from just about every device for every event –infrastructure, endpoints, and security.
  We need to find a better way to get eyes on these logs and do it in a way our SMB-focused client base can afford. To do that, we need to be able to bring them into one system that understands each of the required logs.

• It must be multi-tenant.
  At the time, I had no idea how much doubt I would have in AI until after I watched the various XDRs run. Be ready with a smart team.
  We compared one to another, performing A|B testing while using FortiAnalyzer and raw log data in our Lucene stack as baselines

• Ideally, the XDR must accommodate any vendor, not just those built by the XDR vendor.
  Some XDR vendors we looked at built their own AV, IPS, etc. Others OEM’d someone else’s but wouldn’t discuss it.
  Regardless, I want to know that the tools built into the XDR are mature and tested.

• ***If there’s a cloud component, I want proof that their cloud environment is secure.***All of our clients’ vulnerability data will end up residing there. I don’t want a data breach in our XDR vendor leaking customer vulnerability information. From an espionage perspective, this is an AMAZINGLY rich target. It MUST be safe.
  We evaluate the backend security of all of our vendors. When we did this during our search, one XDR vendor had an amazing product but offered services in a cloud environment had never been security tested!
  Compliance is good, but more importantly? Walk me through how you protect data. Make me feel comfortable that you have taken the measures to protect the data. I was surprised by more than one who couldn’t do this.

• The price structure must be 100% predictable. Variable costs kill. I wanted to make sure we weren’t going to have any surprises. If an XDR vendor asks you, “How many endpoints do you have?” RUN. The pricing structure must accommodate our ability to build it into our subscription costs, at a reasonable margin.
  In the MSSP world, SOC costs can make us fail faster than anything else. How does an MSSP scale without breaking the bank on increasingly expensive information security labor costs?

Our search for the Cinderella XDR (the one that fits us perfectly!):

We looked at dozens of vendors - you’ve heard their names. after nearly two years of competitive analysis, demos, and trials from nearly a dozen XDR companies, we narrowed our focus to two, both undergoing trials, with Stellar Cyber winning us over.

This was a significant capital investment for us. We wanted to make sure we did this right and were able to recoup our investment in added volume and efficiencies. Rather than going with their cloud version, we purchased the 88-core, 20Tb server. The system is designed to parse and analyze vast amounts of data from dozens of infrastructure devices, endpoint logs, and security systems. We wanted it protected, so we racked it up in our secured facility in Iron Mountain Datacenter and performed our first ‘eat your own dog food’ trial during the early summer of last year.

We have MANY lessons learned. I won’t be able to share them all in one short paper, but I thought it might be good to share a few of the bigger ones.

• XDR offers a wonderful solution for bringing just about any piece of information that you can imagine into one pane of glass. We found it overwhelming.

• This is not an entry-level tool. XDR can introduce ambiguity where none should exist. You’ll need a smart team to evaluate every XDR hit before activating SOAR. While the AI learns from the XDR vendor’s larger customer base, it also learns through actions performed by your analysts. They need to be smart.

• Most XDR solutions want to price by the endpoint. This is a deal killer. If a salesperson asks, “How many endpoints do you have?”… RUN.

XDR offers a wonderful solution for bringing just about any piece of information that you can imagine into one pane of glass. We found it overwhelming.

XDR is a fantastic idea, but bad execution will ruin your day. IT guys want to immediately throw everything (and the kitchen sink) at this magic box. And while I fully understand the geek desire for ‘more data is good,’ it made the training curve for our SOC analysts brutally hard.

These things will consume just about any amount of data you can shove into them. We recommend against putting more than one stream into it at a time; at least until you get used to what the machine is going to spit it back out. Why? The machine will produce results on its own, based on preset rules. You’ll find that some are good, but not all -and there will be a lot of them. Your SOC analysts have to know better, They will initially, have to slog through every single alert to verify and validate -did the XDR call it? Was it wrong? What action(s) must be taken? AI, Automation? The magic box? All good things, but without a solid underlying knowledge of what the machine calls good and bad, you could find yourselves overwhelmed. We did. There’s a lot hidden in the black box. Go slow. Let your analysts learn. Bring in one data stream at a time.

Stutzman’s recommendation: Speed kills. Go slow. Start with one data feed. Get it normalized, then add the next.

Know this. XDR is not an entry-level tool.

I take a few SOC shifts every quarter to keep my skills sharp. It keeps me in touch with my SOC, and maybe I do it because it’s one of my favorite jobs! Anyway, during my first shift with a new operational Stellar in our first XDR client, I found myself (at about 2 AM) looking at internal activity behind the firewalls but clearly on the network, with an alert telling me that clear text passwords are being passed at high volume, to fifteen different systems. This bank was not open at 2 AM.

I thought there were only two possible explanations: compromise or vulnerability scanning. As it turned out the client was running OpenVAS to test our response (we passed!), but… how did we see it? I’m looking at internal data from places we’d not seen before! We were now capturing Windows logs, infrastructure logs, authentication logs, and network flow from the 60-person bank. We were pulling in nearly 40 GB of logs per day. I felt like Mr. Magoo, who finally got good glasses and was seeing color for the first time!

As we fully integrate, we’ve retained our FortiAnalyzer and Lucene Stack to allow our analysts to step away from the XDR environment and see data presented in a way they’re familiar with. We’ll do a parallel cutover at some point when old licenses expire. However, as we transition, our Tier 1 analysts (triage analysts) are being forced to learn deeper skillsets. Triage will likely be a thing of the past as XDR takes on automated actions for more mundane tasks like blocking a new scanner or validating the findings of tools before escalating them for action.

Stutzman’s recommendation: Your analysts need to be smart enough to understand what is happening in the data before the AI and Automation take over and the new machine implants mistakes. I’m sixty years old and have been doing this for a long time, but I still wanted a second set of eyes. This is not an entry-level tool. It’s an expert-level tool.

Most XDR solutions want to price by the endpoint. This is a deal killer.

If an XDR vendor asks, “How many endpoints do you have?” RUN!! Endpoint counting doesn’t work in XDR pricing. You will not like the surprise. I cannot stress this enough.

We learned this the hard way. Nirvana for an MSSP is having data from multiple devices on one pane of glass. We installed Stellar Cyber operationally last summer for our own internal operations. We believe in “eat your own dog food” before going live with sales (we use everything we sell).

I asked my IT Director to go one step at a time. Place one information flow into the system, and let’s see how it normalizes. Unfortunately, following our vendor’s lead, he put a span port in our core switch and pushed everything we had into Stellar. The firehose came alive. The XDR generated pseudo flow for over 40,000 devices. Every IOT, mobile, computer, server, every single device with an IP address sitting behind one of our firewalls, anywhere in our client portfolio, was now counted as an endpoint. Our sales team was great. We didn’t get charged while we figured out how to normalize, so we turned off the firehose and started with one client at a time, starting with our own infrastructure. We didn’t want to lose fidelity, so we ended up doing a volume license based on amounts of data, not numbers of endpoints.

Stutzman’s recommendation: Ask for this upfront, then throw as much as you want to at it.

We’ve had our system in for nearly a year now, first as the proof of value starting last March, then going operational during the summer, and now fully operational, deploying it in support of the many NIST 800-171 related projects that we’ve been involved in, and where we’ve got clients who have heterogeneous environments. It’s done a great job. Are we 100%? No. We still require parsers to be written for tools that aren’t already available. We have not yet fully turned on SOAR, and frankly, I’m hesitant to do so in some of our more fragile customer locations where we don’t know what kind of ripple effect the automated action(s) might take.

Are we happy we bought into XDR? Absolutely. The system costs about the same as a couple of good analysts, but I’m confident that it will allow us to scale into clients that we previously would not have been able to access.

Sharing is caring. We had some hard lessons learned and some scary budgetary moments when I thought we were going to have to write some big checks to pay for this thing for —more money than I’d have in our account for a year. Our Stellar team has been amazing, even though we’re a small fish in their bigger pond. I hope this was helpful as you consider your own XDR purchase. Or, if you prefer, contact us. We’d be more than happy to create your XDR in our new multi-tenant Stellar Cyber environment.