Security, Responsibility, Money, and Theatrics
A deeper look at the SolarWinds hack, who is responsible, and what's next?
I hope you all are doing well and welcome (if you aren’t new then again) to Dozen Worthy Reads. A newsletter where I talk about the most interesting things about tech that I read the past couple of weeks or write about tech happenings. You can sign up here or just read on …
This week’s is a long form article on the SolarWinds fiasco and the impact of this. If you have been living under the proverbial rock, about a couple of months ago, FireEye warned everyone that they’d been penetrated by the Russians (and now US intel agencies have officially placed the blame on Russia). This hack impacted several large companies and critical government entities as well) and the true total impact of the hack will first of all not be known for years. This hack essentially was what is known as a supply chain attack.
Why is this a big deal?
Well let’s start with what SolarWinds does first. SolarWinds is a provider of IT management software. From their website:
However this hack was targeted at the Orion Platform update. Hackers inserted malicious into an Orion update which affected 18k customers since they installed that tainted update. On a page on its website that was taken down after news broke out, SolarWinds stated that its customers included 425 of the US Fortune 500, the top ten US telecommunications companies, the top five US accounting firms, all branches of the US Military, the Pentagon, the State Department, as well as hundreds of universities and colleges worldwide. So you can see that this is a huge deal! In fact it is such huge deal that SolarWinds hired Chris Krebs and Alex Stamos as consultants to and I quote “ “be transparent with our customers, our government partners, and the general public in both the near-term and long-term about our security enhancements.” SolarWinds new CEO, Sudhakar Ramakrisha penned this post on the action items they plan to take. As a side note Krebs was fired via a Twitter post by Trump. No relation to Brian Krebs from https://krebsonsecurity.com/ (yep I thought they were related, too!)
Anywho from Ramakrishna’s post …
As we seek to evolve SolarWinds into a company that is “Secure by Design” our internal efforts are focused on three primary areas:
Further securing our internal environment
Enhancing our product development environment
Ensuring the security and integrity of the products we deliver
Now this that context, and to be honest as I started writing this post it felt almost like I was writing some new age spy novel (and sorry to disappoint but no I am not). What I would like to rather analyze is who is responsible for the attack, could this have been prevented? Should this be a punishable.
Thinking back about 5-10 years ago when I was a DB Engineer and this used to be my job, I honestly have to say that how ever much we did we always had unpatched software (and we did a LOT of work all the time). Production systems have unlimited downtimes and it was not always easy to convince the business that we should fix a security issue. To them, it ain’t broken. Why fix it? The end result was we probably had a bunch of Common Vulnerabilities and Exposures (CVE db). This is the first problem
From SolarWinds hack officially blamed on Russia: What you need to know:
How did hackers sneak malware into a software update?
Hackers managed to access a system that SolarWinds uses to put together updates to its Orion product, the company explained in a Dec. 14 filing with the SEC. From there, they inserted malicious code into otherwise legitimate software update. This is known as a supply-chain attack since it infects software as it's under assembly.
It's a big coup for hackers to pull off a supply-chain attack because it packages their malware inside a trusted piece of software. Instead of having to trick individual targets into downloading malicious software with a phishing campaign, the hackers could just rely on several government agencies and companies to install the Orion update at SolarWinds' prompting.
So basically what happened was they probably used some CVE on some system (old OS/DB/network software/open TCP port/dumb password) to access the physical box and inserted code into the “update”.
Let’s think about that for a second. I recall for most of the time that I had to do this we were forced by the security teams at companies I worked at to patch as soon as we could. So if we were in a hypothetical situation where we used Orion, and we had installed the updated, well we’d be screwed too
That brings me to responsibility and accountability which I think is super important. There are multiple players here:
The company making the software
The employees of the company making the software
The investors of the company making the software
The company using the software
The employees of the company using the software
The investors of the company using the software
So who is responsible for what and should it be a crime?
Before we get into that however let’s talk about the Tragedy of the commons. From Wiki :
The tragedy of the commons is a situation in a shared-resource system where individual users, acting independently according to their own self-interest, behave contrary to the common good of all users by depleting or spoiling the shared resource through their collective action.
Take an example of pollution for example. Large companies pollute the roadways with their trucks in the self-serving interest of making more $ but we all have to deal with the pollution.
As Matt Stroller has in, How to Get Rich Sabotaging Nuclear Weapons Facilities, mentioned in general cybersecurity is exactly that:
Indeed, cybersecurity risk is akin to pollution, a cost that the business itself doesn’t fully bear, but that the rest of society does. The private role in cybersecurity is now brushing up against the libertarian assumptions of much of the policymaking world; national security in a world where private software companies handle national defense simply cannot long co-exist with our monopoly and financier-dominated corporate apparatus.
The most interesting part of the cybersecurity problem is that it isn’t purely about government capacity at all; private sector corporations maintain critical infrastructure that is in the “battle space.” Private firms like Microsoft are being heavily scrutinized; I had one guest-post from last January on why the firm doesn’t manage its security problems particularly well, and another on how it is using its market power to monopolize the cybersecurity market with subpar products. And yet these companies have no actual public obligations, or at least, nothing formal. They are for-profit entities with little liability for the choices they make that might impose costs onto others.
This has been happening for decades in the software business where at one point it became a selling point. Case in point : Unbreakable Linux from Oracle.
Lets take each of the constituents one by one
The company making the software
Apparently a security researcher alerted the company last year that “anyone could access SolarWinds’ update server by using the password “solarwinds123.’”. Furthermore the company ran their engineering team in a cheaper location (Eastern Europe) where of course it was probably much easier to break in. Not to impugn on their skills, but its unclear what the code review processes were. BTW, if I am honest here we had SO MANY passwords that were easily hackable. We depended on perimeter security. The network would protect us, then a server would protect us and then and only then would it be possible to hack into a DB with those uh lax credentials. Thankfully never happened
Apparently, as New York Times noted a security advisor warned the management team to take a proactive approach to internal security but they did not pay heed. They also, after the hack, didn’t remove the compromised software for several days after it was discovered. In a larger company such as SolarWinds (~3k employees per Owler) there were probably so many discussions before the decision to remove the software was approved.
Furthermore, in general, security is considered a cost center. It costs money to hire security engineers and in a lot of ways it is like insurance. If you don’t have it and you are affected you are in a lot of trouble. If you don’t have it and nothing happens you look smart. So by underinvesting in security the execs at SolarWinds basically hedged and they lost.. Badly ...Minus mid 2020 their stock price had reached a click before falling precipitously down!
The employees of the company making the software
This one is a tough one but i’d wager that the execution level employees were probably either extremely busy, or were done fighting this battle, or worse still just order takers. I’d like to think it isn’t the last. So now the real question is if the executive leadership was negligent (no we need to move fast, software is good as is) or if the engineer just did some shoddy work should the company hold them responsible? I mean this is going to cost the government years and probably 100’s of millions of dollars to fix? Shouldn’t someone pay for it? Or was the company under investor pressure?
The investors of the company making the software
Companies are in the business of making money for their shareholders. In general this is fine and dandy when skirting the issue if I can call is that is ok as long as things are great. When the proverbial s**t hits the ceiling heads start to roll. Solar Winds was founded in 1999 at a time when the key thing that mattered to stakeholders, be it employees or shareholders was profit. Things have changed of course over the course of time and mission driven companies are where employees tend to hover toward. Personally, if you ask me, never say never, but I like to think that I’d never want to work at a company like Solar Winds (touchwood!)
Anywho, Solar Winds investors are PE firms. PE folks are super smart (I took a class at Berkeley called Turnarounds from Peter Goodson who was a PE guy), but financial engineers. Their key contribution is leveraging debt to buy companies, cutting down costs (improving bottomline) as well. Needless to say they look for things they can optimize and then either re-IPO the company or sell to someone else. As Matt Stoller has written “They are essentially are super-sized mobsters who burn down businesses for the insurance money”
In this case SolarWinds is owned by a person named Orlando Bravo (Thoma Bravo). The key operating tenet for Thoma Bravo is identifying companies with entrenched software/customers and “turning” them around. With the money saved, they either acquire, consolidate, and grow larger. Part of the formula (and one that has played out well) is offshoring as well at outsourcing both - employees and contractors - to cheaper locations. This helps with two things 1) Reduced Costs and 2) Increasing prices since the market is more consolidated. This is to a T SolarWinds!
Founded as a software company
Went public in 2009
PE’d in 2015, a mere 5-6 years ago with $2B debt
More M&A under Thoma Bravo and of course the FTC and the ECC(European Competition Commission) didn’t bat an eyelid. In fact the FTC approved yet another acquisition by SolarWinds in October 2020
In general acquisitions are not accretive. We all know this. But what acquisitions do in such cases is they try to combine the SG&A (Selling, General, & Administrative) costs. Such things like recruiting, HR, Sales, Accounting etc all generally fall under one bucket and products are viewed as pure profit centers with some target Internal Rate of Return (IRR). If the product (or product line) does not meet that IRR the product is killed. So from the outside all this looks brilliant and keeps strengthening their moat and deepens the relationship with the customer (Hey do you know that we also sell databases or blah blah)
With great power comes great responsibility but I guess accounting profits were more important.
The company, employees, and investors of the company using the software
IT professionals really don’t have much of a say. Like I mentioned earlier so many IT professionals are too busy and even if they know there is a problem the business doesnt really care. In many ways it is super frustrating! I knew customers who’d be running 10+ year old DB that wasn’t ever patched. Jeez. Really hard to hold this bunch accountable plus don’t companies buy software and have a reasonable expectation?
What’s next?
A massive hack like this was not only inevitable but also this isn’t the last one or potentially even the only one. There are many such PE firms with many such “hidden” companies that you’d never hear of that cranks the bits and bytes for the products that we use. This will effect not only infrastructure software but also application software across a gamut of verticals. The PE firms get richer, the world gets less secure and we all move on.
I personally think the company should be held liable but this is not going to happen without regulation. There have been so many data breaches of different types - Privacy, Data, Security - in the last 10 years that it's no wonder that Security products are getting a lot of investment. The only way to address is with clear regulation that holds a company accountable with more than a slap on the wrist. For example, Equifax agreed to pay $700 million to settle federal and state investigations into how it handled a massive data breach that affected nearly 150 million people so that means your data or my data is worth about 5 bucks. The cost of a freaking latte! Boeing’s software killed people and they paid $2.5B to settle charges after the 737 MAX crashes
From a today’s regulation perspective Bravo is probably not liable for the hack but Financial Times had an article where Thoma Bravo sold a $315M stake in SolarWinds shortly before the hack was leaked. Now that should be a liable offense since The transaction reduced the exposure of Silver Lake and Thoma Bravo to the stricken software company days before its share price fell. Obviously both Bravo and Silver Lake deny that they knew of this hack when they sold (ok i'll stop here, please laugh out loud). This is nothing but a clear case of insider trading!
What we can expect more of in the future as companies realize that the PR nightmare and fighting lawsuits gets expensive and ultimate long term damaging, Security becomes part of product development versus an afterthought. Security, Data Science, and Analytics get embedded into product to make things more secure. We can also expect more court cases such as this one : Court Rules Deliveroo Used 'Discriminatory' Algorithm.
I think, however, that this is not rock bottom just yet and there will be at least a few such significant breaches and a decade of work and regulation before things are better. The thing with hackers though is that they seemingly always find a way. Expect this war to go on for many decades to come!
Interesting articles about this topic if you want to dig into more detail
Chris Krebs and Alex Stamos have started a cyber consulting firm
Nissan source code leaked online after Git repo misconfiguration
DOJ: Boeing agrees to pay $2.5 billion to settle criminal charges after 737 MAX crashes
Thank you for reading. Stay safe, be well! If you enjoyed reading this please consider sharing with a friend or two (or sign up here if you came across this or were forwarded this)