The security community has been abuzz with an absolutely shocker of story from Bloomberg. The piece reports that the Chinese Government had subverted the hardware supply chain of companies like Apple and Amazon, and installed a ‘tiny chip’ on motherboards manufactured by a company called Supermicro. What the chip did — or how it did ‘it’ was left mostly to the readers imagination.
Supermicro’s stock price is down a whooping 50%, which goes to show just how credible Bloomberg is as a news organization. But besides the Bloomberg story and the sources (all of which are un-named), no one else has come forward with any evidence to corroborate the piece. Instead, both Apple and Amazon have vehemently denied nearly every aspect of the story — leaving us all bewildered.
But Bloomberg are sticking to their guns, and they do have credibility — so let’s wait and see. For now, let’s put this in the bucket called definitely could happen, but probably didn’t happen.
I can only imagine how hard it must be to secure a modern hardware supply chain, but the reason for this post is to share my experience in some supply chain conundrums that occurred to a recent project of mine.
I operate (for fun) a website called GovScan.info, a python based application that scans various gov.my websites for TLS implementation (or lack thereof). Every aspect of the architecture is written in Python 3.6, including a scanning script, and multiple lambda functions that are exposed via an API, with the entirety of the code available on github.
And thank God for GitHub, because in early August I got a notification from GitHub alerting me to a vulnerability in my code. But it wasn’t a vulnerability in anything I wrote — instead it was in a 3rd-party package my code depended on.
Dependency Hell!
Modern day programs aren’t just one ‘block’ of code written by one person (or even just one thing). They’re a many blocks of code coming together to achieve a specific task. In my case, I was running a Python script that required a Python Interpreter (obviously!) — but above and beyond that I was depending 28 different Python packages written by people who were not me or Python.
These included packages to help me integrate to AWS, to call Shodan for their results, to extract TLD data from a domain, and the list goes on Instead of writing all these from scratch, I decided to import existing python packages that were already written (mostly by people smarter than me), and this tremendously improved the time-to-market of the end solution.
For example, you should never try to write your own Python Package to call the Amazon API directly — instead use Boto3, it’ll be safer, quicker, more secure and you’re guaranteed it’ll be maintained for a long time.
But I was also depending on code that wasn’t written by big companies like Amazon. Some were written by independent folks, people like me, just coding something over the weekend. How could I be sure that these folks aren’t malicious — or at the very least keep their code up to date and free from vulnerabilities. (spoiler alert: I can’t be sure)
And here’s the real kicker — some of the packages I depended on, had dependencies of their own!!
A single import statement in my code, could include multiple dependencies that originated from dozens of people, all of whom may not even know each other — and a single malicious actor in that chain could compromise my code and any other that used the same packages. And to me it would look like one dependency — but in reality there could be dozens of dependencies hidden underneath it.
In my case, the vulnerable package was cryptography, which I didn’t import directly, but rather was imported by sslyze, a separate package I used to extract certificate information from websites. I didn’t even know I was depending on cryptography, but fortunately GitHub picked it up from the requirements.txt file. Which thankfully I kept up to date (after all this is how I deploy the app)
It didn’t take me long to upgrade the package, re-test and re-deploy, but next time I might not be so lucky. GovScan.info is just a side-project of mine, so this wasn’t a big deal.
But imagine a worst-case scenario
You’ve got a Production running script your company relies on for a core business operation. Today, a high severity vulnerability was reported in a python package that 70% of your code base relies on — and to make matters worse, the developer of the package and since stopped supporting it (nobody is going to fix this for you!). Ripping out the package from the python script won’t be easy, and worse still, you’re a small operations with no automated testing.
You’ve got 3 options:
- replace the package with another similar package, re-code and re-test
- fork the package and try to fix the vulnerability, supporting the package indefinitely from now on
- Find some clever way to limit the risk of the vulnerability to an acceptable level.
In the short-term your choices are clear, either live with the vulnerability for 6 weeks, or shutdown a core business process. Guess which one management is going to gravitate towards?
And devoting developer time to do this, will severely impact the timelines of another critical project.
With all of that, it’s clear that using un-supported packages is not a good idea — but that’s what people do all the damn time! Most folks aren’t even aware of their first-level dependencies, let alone their 2nd, 3rd of 4th level ones. And the example is Python, but you can replace with Java, C, PHP or whatever other language you choose, and it would still hold valid.
If this was all a vendor product, it’ll be moot — you can just choke the vendor (assuming you know what packages they rely on — or hopefully the vendor knows). In-House built applications though may not go through this level of scrutiny, or perhaps only go through this on an ad-hoc ‘as-you-deploy’ basis. Legacy applications written in 6 year old versions of Java may be riddled with vulnerable dependencies but still run happily in production until someone comes along for a code-base scan.
Plus, this is not a binary outcome, just because you use a vulnerable package doesn’t mean you’re vulnerable, in many cases you’ll need to use the package in a specific way to be vulnerable — and that requires some human intervention to really figure out.
And we’ve not even begun to explore the other stuff running on the server, like what OS are your running, what version JRE or .NET, what version of that logging application, that monitoring tool, that little batch script that bob wrote before he left?
Has anyone even looked at the hundreds of excel macros littered in the organization? Whose validated that the Excel VBA Rebecca wrote two days before she quit, doesn’t exfil data to pastebin?
The point is, supply chain issues are a massive problem — it’s tremendously hard with hardware, but even software supply chain problems (with theoretically should be easier) is immensely hard.
Conclusion
The Bloomberg isn’t a cause for alarm — yet!
There’s probably a hundred easier ways for China to hack into you, without the need for specialize hardware — and even if they needed the hardware, it’ll probably be easier hijacking a local laptop than a server that maybe behind a massive amount of firewalls and NAT-ed routers.
What’s probably more pressing is the software supply chain, especially in custom built apps, most media outlets I know use WordPress and install a boat-load of plugins on them — whose checked any of those for vulnerabilities lately?
In any case, taking a fine tooth comb to your software supply chain probably couldn’t hurt — but it won’t be an easy task.