The long hack: how China exploited a US tech supplier

20 Comments

1. 2021-02-12 10:59 pm

   sukru

   More details on the “hack” here:
   https://hackaday.com/2019/05/14/what-happened-with-supermicro/

   Basically it is a hijack of the on board components; not too dissimilar to hacking a gaming console to dupe it into running a custom ROM.

   The issue is these kinds of servers are supposed to be behind a firewall. The important component here is called IPMI and is an “out of band” management system. Essentially a separate computer with separate network connection and can even run when the CPU or RAM is not installed on the server.

   Basically, it is a backdoor you knowingly put on the system.
   Basically, it should be behind the strongest firewall you can have.
   Basically, it should not have open internet access (in either direction).

   But many people just expose those ports: https://blog.rapid7.com/2013/07/02/a-penetration-testers-guide-to-ipmi/

   Even without any hardware hijacking those are dangerous components.
2. 2021-02-13 3:18 am

   Anonymous

   Bloomberg is not a reliable source. Their editorial policy very clearly states they produce articles to disrupt the market and their reward structure rewards this. Before you read a Bloomberg article you have to have this in mind before you read anything.

   Tabb declined to address specifics of the FBI’s probe. “The Chinese government has been doing this for a long time, and companies need to be aware that China is doing this,” he said. “And Silicon Valley in particular needs to quit pretending that this isn’t happening.”

   Neither Supermicro nor any of its employees has been accused of wrongdoing, and former U.S. officials who provided information for this story emphasized that the company itself has not been the target of any counterintelligence investigation.

   An investigation is not proof of guilt plus there is the case of exactly which story is it? The first one or the second one?

   Throughout, government officials kept their findings from the general public. Supermicro itself wasn’t told about the FBI’s counterintelligence investigation, according to three former U.S. officials.

   The secrecy lifted occasionally, as the bureau and other government agencies warned a select group of companies and sought help from outside experts.

   So now we have a different set of people telling a different story.

   “In early 2018, two security companies that I advise were briefed by the FBI’s counterintelligence division investigating this discovery of added malicious chips on Supermicro’s motherboards,” said Mike Janke, a former Navy SEAL who co-founded DataTribe, a venture capital firm. “These two companies were subsequently involved in the government investigation, where they used advanced hardware forensics on the actual tampered Supermicro boards to validate the existence of the added malicious chips.”

   Janke, whose firm has incubated startups with former members of the U.S. intelligence community, said the two companies are not allowed to speak publicly about that work but they did share details from their analysis with him. He agreed to discuss their findings generally to raise awareness about the threat of Chinese espionage within technology supply chains.

   So we have a meathead and his chums sharing an analysis which is not discussed in any detail and no actual published proof.

   In an unusual disclosure for any public company, Supermicro told investors in May 2019 that its own computer networks had been breached over multiple years. “We experienced unauthorized intrusions into our network between 2011 and 2018,” the company wrote. “None of these intrusions, individually or in the aggregate, has had a material adverse effect on our business, operations, or products.” The company didn’t respond to requests for additional details about those intrusions.

   Now we go from Supermicro being a tool of the Chinese government to being a victim of unknown external threats.

   Another Pentagon supplier that received attention was China’s Lenovo Group Ltd. In 2008, U.S. investigators found that military units in Iraq were using Lenovo laptops in which the hardware had been altered. The discovery surfaced later in little-noticed testimony during a U.S. criminal case—a rare public description of a Chinese hardware hack.

   “A large amount of Lenovo laptops were sold to the U.S. military that had a chip encrypted on the motherboard that would record all the data that was being inputted into that laptop and send it back to China,” Lee Chieffalo, who managed a Marine network operations center near Fallujah, Iraq, testified during that 2010 case. “That was a huge security breach. We don’t have any idea how much data they got, but we had to take all those systems off the network.”

   I’m unclear exactly who is saying this and what their qualifications are and whether they are a primary source or not. It is well known that GCHQ and 77th Brigade et al get up to tricks as well as the NSA et al. It’s also known Lenovo dropped the ball on update patches which had “helpful” undocumented features. I’m trying to discover if the people in this article are discussing an actual built in exploit which does what they say it does or whether they are discussing an external agency exploiting flaws in the hardware and software.

   Lenovo was unaware of the testimony and the U.S. military hasn’t told the company of any security concerns about its products, spokeswoman Charlotte West said in an email. U.S. officials conducted “an extensive probe into Lenovo’s background and trustworthiness” while reviewing its 2014 acquisitions of businesses from IBM and Google, West said. Both purchases were approved.

   “As there have been no reports of any problems, we have no way to assess the allegations you cite or whether security concerns may have been triggered by third-party interference,” West said.

   Quite.

   After the discovery in 2008, the Defense Department quietly blocked Lenovo hardware from some sensitive projects, the three U.S. officials said, but the company was not removed from a list of approved vendors to the Pentagon.

   So which one is it?

   Investigators attributed the rogue code to China’s intelligence agencies, the officials said. A former senior Pentagon official said there was “no ambiguity” in that attribution.

   Officials always say that for enemy of the week.

   The moves allowed America’s own spies to begin gathering intelligence on China’s plans without alerting Beijing, the two officials said.

   Uh, huh. These would be the same mates as meathead, I presume?

   As they investigated the Pentagon’s data centers, government officials took discreet steps to try to prevent the use of Supermicro products in sensitive national-security networks—even though the company remained on public lists of approved suppliers.

   What like Juniper and Cisco? Every other interdicted product? Microsoft Windows?

   As military experts investigated the Pentagon breach, they determined that the malicious instructions guiding the Pentagon’s servers were hidden in the machines’ basic input-output system, or BIOS, part of any computer that tells it what to do at startup.

   I thought we were discussing a hardware problem not a firmware problem? Oh, so now it’s a firmware problem? Yes, we already know BIOS can be exploited and pretty much by everyone.

   “This was espionage on the board itself,” said Mukul Kumar, who said he received one such warning during an unclassified briefing in 2015 when he was the chief security officer for Altera Corp., a chip designer in San Jose. “There was a chip on the board that was not supposed to be there that was calling home—not to Supermicro but to China.”

   What chip? Are they actually sure the chip was modified or was it as per design? Was it the chip itself or was it being exploited by a BIOS or other software compromise. The article does not say.

   “The agents said it was not a one-off case; they said this was impacting thousands of servers,” Kumar said of his own discussion with FBI agents.

   Presumably they have thousands of evidence samples so wouldn’t miss one if they rpesented it as evidence. Where is it?

   “This wasn’t a case of a guy stealing a board and soldering a chip on in his hotel room; it was architected onto the final device,” Quinn said, recalling details provided by Air Force officials. The chip “was blended into the trace on a multilayered board,” he said.

   “The attackers knew how that board was designed so it would pass” quality assurance tests, Quinn said.

   So now we have BIOS explots, IC exploits, possibly software exploits to deliver payloads, and now board level exploits. Board level exploits can be created by taking advantage of a board design or meddling with a board design before production.

   In its written response to questions, Supermicro said that no customer or government agency has ever informed the company about the discovery of malicious chips in its equipment. It also said it has “never found any malicious chips, even after engaging a third-party security firm to conduct an independent investigation on our products.” The company didn’t respond to a question about who chose the samples that were investigated.

   After Bloomberg reported on the added-chip threat in October 2018, officials for the U.S. Department of Homeland Security, the FBI, the Office of the Director of National Intelligence and the NSA made public statements either discounting the report’s validity or saying they had no knowledge of the attack as described. The NSA said at the time it was “befuddled” by Bloomberg’s report and was unable to corroborate it; the agency said last month that it stands by those comments.

   […]

   “What I was told was there was an additional little component on the Supermicro motherboards that was not supposed to be there,” said Mott, who has since retired. He emphasized that the information was shared in an unclassified setting. “The FBI knew the activity was being conducted by China, knew it was concerning, and alerted certain entities about it.”

   So it wasn’t a super secret stealth IC attached to a sneaky track designed in but a dodgy IC attached to an unspecified track on the board? So we’ve gpne from super stealthy sneaker top level hack to a kludge. Which one is it? Where is the proof? Which track? Which IC?

   APT 17 specializes in complex supply-chain attacks, and it often hits multiple targets to reach its intended victims, according to cybersecurity firms including Symantec and FireEye. In 2012, the group hacked the cybersecurity firm Bit9 in order to get to defense contractors protected by Bit9’s products.

   Like the NSA interdiction group or GCHQ meddling? Oh so now we’re getting closer.

   Breaches involving Supermicro’s update site continued after the Intel episode, according to two consultants who participated in corporate investigations and asked not to be named.

   In incidents at two non-U.S. companies, one in 2015 and the other in 2018, attackers infected a single Supermicro server through the update site, according to a person who consulted on both cases. The companies were involved in the steel industry, according to the person, who declined to identify them, citing non-disclosure agreements. The chief suspect in the intrusions was China, the person said.

   In 2018, a major U.S. contract manufacturer found malicious code in a BIOS update from the Supermicro site, according to a consultant who participated in that probe. The consultant declined to share the manufacturer’s name. Bloomberg reviewed portions of a report on the investigation.

   And closer… So somebody hacked a website to upload a BIOS exploit to activate a magic IC they won’t name which is attached to a track on a board they won’t name.

   “If you think this story has been about only one company, you’re missing the point,” he said. “This is a ‘don’t let this happen to you’ moment for anyone in the tech sector supply chain.”

   And closer… The fact is nobody can be trusted. Was it the Chinese? I have no idea. Was it likely? I wouldn’t put it past them but there are other actors who would want to pass off things as the Chinese, or the Chinese would want to pass it off as them, and so on for every actor on the stage. There are no surprises in this article and what is not said is as interesting as what is said. Without proof I’m just going to put this down as a typical Bloomberg disruptive article planted to shift the market for trade reasons. Not that there aren’t concerns but neither are Bloomberg et al whiter than white.
• 2021-02-14 3:25 am

  unix_joe

  >Bloomberg is not a reliable source. Their editorial policy very clearly states they produce articles to disrupt the market and their reward structure rewards this.

  Agreed, 100%. Although this wasn’t apparent to me until reading how they covered last month’s Gamestop pump and dump. The “market disruption” and “financial revolution” garbage articles they were putting out could have been written by any Redditor who just got into the market when they started getting stimulus checks last Spring. And, the now that we know it was rival hedge funds pumping the price all along leaving undisciplined WallStreetBets children holding the bag, not a single retraction or Op-Ed.
• 2021-02-15 12:46 pm

  chowyunpat

  This is been proven years ago that the Chinese hardware and software exploits into the products, not to mention industrial espionage in the technology and scientific sectors. China is also one of, if not the largest lobbying group in Washington DC and it sounds like you could be on their payroll too.
• 2021-02-15 6:36 pm

  Anonymous

  @chowyunpat

  This isn’t about theoretical exploits which every side is both capable of and guilty of but utterly debunked BS posted by Bloomberg not just once but twice. Respected people in the security community have posted pretty much identical teardowns of what I posted so gtfo. (Yes I could have rephrased things a it more carefully but dealing with BS is exhausting and I was being lazy.)

  Twitter thread here:

  https://twitter.com/pwnallthethings/status/1360234953011851264

  Who is behind this Twitter account? Ex British spy and “professor at the University of Texas at Austin’s Strauss Center for International Security and Law, where he teaches a graduate course, “Cybersecurity Foundations: Introduction to the Relevant Technology for Law and Policy.”

  https://www.washingtonexaminer.com/weekly-standard/this-former-british-spy-exposed-the-russian-hackers

  You were saying?
• 2021-02-15 7:57 pm

  Anonymous

  P.S. I don’t necessarily agree with Tait’s opinions and intpretations about the Russian hacks or Snowden leaks. There are differing opinions from a former NSA expert and others but this isn’t a discussion I want to get involved with. It’s far too opinionated area and I neither have the inside knowledge nor want to get involved with arguments as it’s all done to death by now and won’t change anything.
3. 2021-02-13 4:48 am

   hardcode57

   ‘we all know giant corporations and American intelligence agencies never lie.’ Also we all know that journalists never get it wrong.
   That’s the problem with these kind of stories: the journalists aren’t qualified to tell them, and those that are qualified can’t be trusted.
4. 2021-02-13 12:15 pm

   Anonymous

   More from The Register.

   https://www.theregister.com/2021/02/12/supermicro_bloomberg_spying/

   Yet the Bloomberg report today does provide a named source, Mukul Kumar, who as chief security officer for FPGA designer Altera claims to have learned of such a spy chip during an unclassified briefing. “This was espionage on the board itself,” he is quoted as saying. “There was a chip on the board that was not supposed to be there that was calling home — not to Supermicro but to China.”

   And?

   The Register spoke with a former executive at a major semiconductor company who asked not to be named, about the plausibility that the subverted silicon cited in the Bloomberg report might exist and we were surprised to find that he found it credible.

   “I have physically held evidence in my hands,” he said with regard to the existence of compromised hardware. “I have seen it from multiple governments.”

   Apparently this executive pinky swears he saw the evidence.

   Different people some of whom may or may not exist only on paper working at agencies who compartmentalise information on a need to know basis, and lots of people with vested interests either as private sector contractors or business rivals all of whom use vague words about the nature and type of exactly what is being discussed and all of whom assert something which has not been seen by anyone so they can publicly verify the evidence makes me doubt the story.

   Not even GCHQ could find anything wrong with Hauwei kit and if they did they are not admitting it. The head of MI6 (SIS) basically admitted in public and on the record it wasn’t anything with Huawei’s technology but the trade issue with the US which was the “security” issue. <— This.

   Colin Powell's UN presentation? The UK's "dodgy dossier"? Snowden?

   For all we know this entire story may boil down to an open JTAG. Wow. Like nobody does that.

   Plus Bloomberg. Yeah…
5. 2021-02-13 11:18 pm

   cpcf

   All sides of this use trade as a political and military weapon, I’m not sure what people expect.

   From reading a few different reports the Huawei stuff is primarily in retaliation to theft of IP, something that has been going on for decades and decades. I’m not sure much ground can be made when China doesn’t even blink at delivering the death sentence to someone accused of corporate spying.

   It makes perfect sense that an entity like MI6 or GCHQ state nothing to see here, they have been doing that since Enigma was a pup. You never start off a security investigation by pronouncing, “I know what you know!”

   The biggest advantage is gained by knowing without letting the opposition know you know!

   The technical debate is a bit of a moot point.
• 2021-02-14 7:25 am

  Anonymous

  Of course there is the “great game” and supply chain issues are in there somewhere but the whole Bloomberg article whiffs of “buy on the rumour, sell on the news”. Whether the story has any basis in fact or is simply anecdote and hearsay and nonsense remains for the reader to determine but there has been no coherent argument or display of evidence. I disagree the evidence issue is a moot point. I’m not a fan of flimmery and secrecy for the sake of secrecy nor being fed lies because it’s the “patriotic” thing to do.

  I’m fine with the basic security technical arguments and security arguments on balance of trade issues. I’m not fine with lies to cover up politicians embarassment or bad business decisions simply to slide government “corrections” of the market past the WTO.

  As for none headline security issues there’s rather a lot of lying going on now to cover politicians mistakes and embarassment and greed causing any number of fundamental human rights abuses in the UK. Or is this a “moot point”?
• 2021-02-14 7:03 pm

  cpcf

  @HollyB. It’s all rather arbitrary, arguing right or wrong implies some knowledge that in the public sphere of the debate often comes from or is based on the same sources as the information being questioned.

  The absence of evidence isn’t a greater sign of a conspiracy, and a public correction isn’t automatically the sign of some prior evil intent, yet much of the public debate seems to proceed along those lines, leveraged by the likes of Bloomberg and News Ltd.

  I thought the public outrage at the WHO latest proclamation was a great example. Did anyone expect otherwise?

  The irony is that many busy slamming the WHO’s China COVID conclusions also strongly criticise Western Governments for hoarding vaccine.

  I see the WHO as being completely consistent, it’s objective/priority is to protect and repair, and it can’t protect and repair the global disadvantaged without China’s assistance. So is there any doubt China got the all clear, not for any reason other than we need China’s capabilities to provide cheap vaccines and China is happy to politicise that resource!

  So the WHO evidence also becomes a moot point, because the evidence conflicts with it’s objective, it has so many parallels to Huawei and other technical embargoes.
• 2021-02-15 3:43 am

  Anonymous

  @cpcf

  The issue isn’t what can be done technically whether it is hardware or software or some other form of exploit but the issue of nonsense stories and misinformation and misdirection with a large dose of hypocrisy.

  I wasn’t born yesterday and know when I am being flannelled and being taught to suck eggs.

  The UK is currently neck deep in bad governance and human rights abuses and cover-ups. They do this not because that is where the science or evidence or even law is but because they can. There is no other reason.
6. 2021-02-13 11:33 pm

   cpcf

   Just as an aside.

   One of our factory sites has a major cell node for the surrounding micro-cells, each node serves up about a dozen other towers dotted at regular intervals around the horizon. Twice in the past year we have been contacted by sub-contractors claiming to be telecoms maintenance, both times asking for outside hours site access to the cell tower, yet by law the tower owner and operator have their own bylocks and we can’t refuse them 24×7 access.

   The second time we got a request we contacted the tower operators and they said they had no idea about the request and thanked us for not allowing the callers/visitors onsite. When we did our own investigation, the company requesting access did not exist, we haven’t heard anymore and we do not expect we will. These aren’t people that turn up with a ladder and some pliers, they arrive with high reach cherry-pickers and multiple trucks with 3 or 4 workers in each cabin. The give away is asking us for access, the official tower owners never do, they just report when they will be onsite and let themselves in.
• 2021-02-15 4:02 am

  Anonymous

  I worked for a company with connections to GCHQ and we had stuff diverted by the Soviets. This was not a surprise. There is also a fascinating story about the CIA conducting an operation on Russian soil to dismantle and analyse a Soviet space capsule. There are also stories of the Russians obtaining none classified material from a US naval base by which means they were from the dougnut orders able to assess the capabilities and capacities of the naval base. Another story and it’s not the only example of the US opening its big fat mouth about classified UK material.

  These past few years have seen a lot of fingers being pointed and wagged in all directions and that’s just on issues of human rights abuses and nuclear anti-proliferation treaties and the pandemic. There are also countless stories of so-called advanced Western nations (thinking of two in particular) ramming through public policy like the snoopers scharter and “weaponised psychology” and “economic murder” and dodgy tax havens and the list goes on. I have a fair idea what goes on in the rest of the world. Some of it is good and some of it is not very pretty. What does annoy me domestically is advanced nations with no excuses causing problems because of extremist elements which harm their own populations and who respond with the rallying cry we need more pain or a big evil to finger point at. It’s not just the big issues or legacy issues but wholly avoidable petty minded cruelty of it all.
7. 2021-02-15 5:44 pm

   Strossen

   The Bloomberg hack story is just QAnon for a different social demographic
• 2021-02-15 6:38 pm

  Anonymous

  Ooh. That’s a zinger.
8. 2021-02-19 8:55 am

   atrocia

   Bruce Schneier considers the Bloomberg story legitimate:
   https://www.schneier.com/blog/archives/2021/02/chinese-supply-chain-attack-on-computer-systems.html
• 2021-02-19 11:03 am

  Anonymous

  Schneiers gone a bit “Atlantic Council” in his old age. Most of his readers think the story is junk.
• 2021-02-19 12:46 pm

  atrocia

  Well, I’m no expert, and I don’t really have an opinion of my own, but I trust Schneier’s judgment more than I do that of anonymous blog commenters – here or there.
• 2021-02-19 1:15 pm

  Anonymous

  Schneier is correct about two things: Supply chain issues both theoretical and real exist, and military and security services consider it to be a real problem. Nobody has an issue with this. The Bloomberg story is still junk.