Malware, distributed by SolarWinds Orion software updates, infected the networks of the White House, the DOJ, the State Department, NASA, NSA, the military, the top IT and telecommunications companies, and most of the Fortune 500 companies. Foreign governments and private companies have been hit, too. In total, up to 18,000 large entities have been infected by the malware.
The perpetrators of this malware attack were SolarWinds employees, not any outside party.
The idea that the malware was not inserted by SolarWinds employees, but by outside attackers is preposterous and not supported by any evidence. The one and only source of this allegation is FireEye’s blog, which also claims that there is a nation-state behind the attack, without naming it. Remarkably, when filing the special SEC report on the subject, FireEye did not directly repeat this claim, but stated that it is on the company’s blog. SolarWinds, who should know for certain how the company became a malware distributor, refused to directly support this theory, but hinted at it: “On Saturday, December 12, our CEO was advised by an executive at FireEye of a security vulnerability in our Orion Software Platform … While security professionals and other experts have attributed the attack to an outside nation-state, we have not independently verified the identity of the attacker.”
The call that he alleged nation-state is Russia was made by the media without any evidence for purely political reasons. Then these allegations have been repeated in an echo-chamber and circularly referenced.
The motives of the SolarWinds employees behind the malware creation and distribution could be numerous, from opposition to President Trump, to aiding espionage by foreign nation-state(s). Ordinary criminal interest cannot be excluded.
If a foreign nation-state was involved, Russia is an unlikely culprit. The suspicions against Russia within the cyber security circles are strong. Russia has relatively little leverage over the tech companies in the US. Additionally, SolarWinds develops its products and/or provides support from countries, which are difficult for Russia to infiltrate (including Singapore and Philippines). The Russian government denies any involvement.
SolarWinds and its Malware
On Sunday, December 13, SolarWinds announced (SWI 1 – see the Annotated References) that updates to its leading network management software Orion, shipped to customers from March 2020, contained malware. As a network security and administration tool, Orion had access to all network services, servers, and other computers where it ran and had Administrative access. Upon installation, the SolarWinds malware calls its command-n-control (C2) server, apparently not under the control of SolarWinds, and downloads instructions. When needed, it downloads additional pieces of malware and can install and spread them all over the network. One type of malware, computer viruses, can infect innocent software, which infect other networks, and so on.
As a network security company, SolarWinds is required to have strong security protections. It is required to keep logs of network activities and all changes to its production build, which are linked to the individuals who made these changes. According to statements on the company website, SolarWinds these and other precautions are in place.
SolarWinds (SWI) is a publicly traded company with market caps of about $5B (December 22). SolarWinds has 3,200 employees, most of whom are located abroad. SolarWinds also claims customers in more than 190 countries, including China. SolarWinds is headquartered in Austin, TX.
The perpetrators started preparations for the distribution of the malware in October 2019. Then they built and pushed to production updates with actual malware, from March to June 2020. These trojanized updates were pushed for 2019 and 2020 versions of SolarWinds Orion through at least October 30, 2020. Security Scorecard also reported that the updates were still available from the SolarWinds website as late as December 18.
FireEye
FireEye (FEYE) is a publicly traded company with market caps of about $5B, coincidentally close to SolarWinds’ (December 22).
FireEye has made a business out of the false attribution of network breaches to state actors. In this respect, it is second only to CrowdStrike. In 2016, FireEye supported the false claims that Russia hacked the DNC and leaked its dirty laundry, made by CrowdStrike.
The circumstances in which FireEye made the claims that SolarWinds was hacked by a nation-state are also very unusual. On December 8, FireEye made a public announcement that it was hacked by an unnamed nation state. The company then skillfully avoided directly repeating this claim in the same day SEC filing and instead only referenced the announcement on their company blog: “On December 8, 2020, concurrently with the filing of this Current Report on Form 8-K, FireEye, Inc. (‘FireEye’, ‘we’, ‘our’ or ‘us’) is announcing on our corporate blog that FireEye recently was attacked by a highly sophisticated cyber threat actor, one whose discipline, operational security, and techniques lead us to believe it was a state-sponsored attack.”
FireEye claimed that the intruders stole its “Red Team” (i.e., hacking) tools. Hacking tools are not a rarity. They are widely available as network testing or Red Team tools, not only on black market, but also from legitimate vendors. FireEye did not explain why the “highly sophisticated cyber threat actor” nation state actor would need its hacking tools. Anyone capable of gaining access to the SolarWinds malware would have no need for FireEye’s hacking tools. FireEye also did not say when the alleged breach happened.
If such a breach indeed happened, it is very unusual for a network security company to publicly spotlight it. FireEye could have quietly published an advisory saying that their Red Team tools have been accessed without authorization with the usual “we are investigating … have no knowledge … reporting out of abundance of caution … our mission is to blah-blah-blah, and the intruders will not stop us …”.
A few days later, on December 13, FireEye announced that its network was breached through malware in SolarWinds Orion, and that SolarWinds was also breached by the same nation-state actor! Even if this were true, how could they know? The malware was in SolarWinds’ software, received from SolarWinds, and signed by SolarWinds’ security certificate. Everything in the malware points to it having been built by SolarWinds. A logical deduction would be that some employees of SolarWinds maliciously added the malware code into the build. Only an insane flight of fantasy can lead to the idea that some outside third party breached SolarWinds’ network (which is harder to do because SolarWinds is a network security company), got access to its software signing security certificate, source code, production build and deployment configuration, inserted the malicious code, and somehow made everyone in the quality assurance and customer support departments fail to notice any issues for eight months.
According to both FireEye and SolarWinds, FireEye informed SolarWinds that it is aware of the malware in its Orion updates on December 12. SolarWinds is the primary suspect in this incident, threatening the national security. Apparently, FireEye informed SolarWinds before informing its own customers, for whom it provides network security services. This looks more like a criminal collusion, than security.
WaPo and NY Times
Russia was first named in the Washington Post and the New York Times on December 13, on the same day that FireEye and SolarWinds announced the alleged hack. The WaPo article (the first version of it) was written by Ellen Nakashima, the same writer who “broke” the fake news that the DNC network was breached by Russia in June 2016. The NY Times article was written by David Sanger, another full time Russia hoaxer. He literally wrote a book on it. Having previously reported such misinformation discredits both these writers from being trustworthy journalists.
SolarWinds Malware Perpetrators
It has been now discovered that a dummy payload was inserted into the SolarWinds Orion in October 2019 (MSFT 1), but a real malware was inserted only in March 2020. Outside “threat actors” would have exploited the access instantly because they would run the risk of it at any moment. Only insiders could afford to wait five months, assured that they would not lose their access. This delay clearly points to the culprits being insiders.
The malware was added to Orion by SolarWinds software engineer(s), responsible for the production builds. Build configuration files contain tens of thousands of lines and/or include hundreds of other build files. If SolarWinds has the level of security and discipline which it promises on its website, one person cannot do that. A collusion of two or more engineers in R&D and QA is required. The malware is sophisticated, as would be expected from employees of a security software company, but one part of it is not. It is the DNS call to resolve the address of the C2 server. It has the following format:
3mu76044hgf7shjf.appsync-api.eu-west-1.avsvmcloud[.]com
The traffic between malware and its home is usually encrypted and is expected to use steganography. But DNS requests are visible to traffic management software and saved in firewall logs. There is a myriad of products to analyze such logs for suspicious requests. Any new domain is suspicious. avsvmcloud[.]com domain was anonymously registered and recently (on February 27) moved. Its prior history was short and anonymous (2). This triples the suspicion. A request coming from a server which has no reason for making such requests should ring alarm bells. Additionally, there are two other attributes of maliciousness: a) subdomains like eu-west-1.avsvmcloud[.]com are intended to look like a large scale cloud services, and b) the long random string on its left side is rare for legitimate subdomains. Many log analysis tools use AI to flag such requests. An average network security person, reviewing flagged entries, would recognize these requests as malicious. This malware was installed in 18,000 organizations with high cyber security awareness, many of which having many sites. Undoubtedly, dozens or hundreds of cyber professionals noticed and contacted SolarWinds support. Support is a separate department from software development and QA. If the malware were planted by an outside attacker, SolarWinds would detect and eliminate it within weeks based on the customers’ concerns. This did not happen, and the malware remained undetected for nine months
Thus, the perpetrators had accomplices within SolarWinds support department. The job of this personnel was to assure customers that this is fine and prevent the issue from escalating to the upper management (if it was not involved). SolarWinds has customers all over the world and across many verticals, and many customers contact their salesperson rather than technical support.
One might guess that some cybersecurity professionals in the defense, national security, and/or law enforcement went not to SolarWinds, but to the government agencies responsible for cyber security, like the CISA and the FBI. Somebody had to be there to bury their concerns. Thus, if somebody hacked SolarWinds, they also hacked the CISA and/or the FBI.
Remarks
It is not a supply chain attack. In a supply chain attack, attackers (A) attack their target (T) by compromising target’s vendor (V). IIn our case, SolarWinds is both the vendor and the attacker. It can be legally disputed, if the company was not aware of what its employees were doing, and some other conditions. Unless SolarWinds cooperates with the law enforcement out of the public sight, this is not the case.
So far, the story is remarkably similar to what happened in 2016 with CrowdStrike’s attribution of the DNC leaks to an alleged Russian hack.
SolarWinds owners Silver Lake and Thoma Bravo sold ~ 6% of their stock to Canada Pension Plan Investment Board on December 7. They deny prior knowledge. This is likely not a factor.
It is hard to understand why was SolarWinds, a transnational company with most employees abroad and customers all over the world, was entrusted to secure extremely sensitive US networks? Just the fact that its software is available all over the world allows foreign adversaries to find vulnerabilities in it. Finding a vulnerability is harder than exploiting a known vulnerability.
The source code for the software deployed by DoD, for example, used to be inspected. When did they stop this necessary practice?
Annotated References
(1) A reader’s comment in KrebsOnSecurity (not connected to CISA’s Krebs), Dec 18
“Anyone looking at their firewalls? This is plain incompetence with Krebs (CISA director) at the top of the heap of idiots. Yeah looking at firewall logs is eye-wateringly boring but ITS YOUR JOB!”
(2) DomainTools Post on SolarWinds and Sunburst, Dec 14
“1. Primary C2 infrastructure was acquired and initially provisioned in December 2019 using an existing but likely expired domain that superficially resembles a domain for cloud hosting services.
- Hosting, name server, and other items necessary for operational use were established on 27 February 2020.
- If avsvmcloud[.]com is the only primary C2 domain, the SUNBURST campaign was operational no earlier than 27 February 2020, with a significant change in aspect on 30 October 2020.”
This article delineates the history of the domain avsvmcloud[.]com, which has been hosted in Microsoft cloud since February 27, until it was seized by Microsoft. by legal process.
(3) Cloudfare’s blog post, Dec 16
Shows dynamics of requests to the SolarWinds malware infrastructure, timeline, maps, and animation.
SWI
(SWI 1) SWI filing with SEC, Dec 14
“SolarWinds Corporation (“SolarWinds” or the “Company”) has been made aware of a cyberattack that inserted a vulnerability within its Orion monitoring products which, if present and activated, could potentially allow an attacker to compromise the server on which the Orion products run. SolarWinds has been advised that this incident was likely the result of a highly sophisticated, targeted and manual supply chain attack by an outside nation state, but SolarWinds has not independently verified the identity of the attacker.”
(SWI 2) SWI filing with SEC, Dec 17
“On Saturday, December 12, our CEO was advised by an executive at FireEye of a security vulnerability in our Orion Software Platform which was the result of a very sophisticated cyberattack on SolarWinds. We soon discovered that we had been the victim of a malicious cyberattack that impacted our Orion Platform products as well as our internal systems. While security professionals and other experts have attributed the attack to an outside nation-state, we have not independently verified the identity of the attacker.”
SolarWinds is in the best position to determine the nature of the “attack,” and has certainly made that determination (or a decision not to make it) by Dec 17. This language indicates that they are aware that FireEye is lying.
“The vulnerability has only been identified in updates to the Orion Platform products delivered between March and June 2020, but our investigations are still ongoing.”
This is not true. People reported that they were able to download the trojanized (infected with malware) Orion updates even on Dec 13-14. They might have meant that the “updates were built between March and June 2020.”
“After our release of Orion 2020.2.1 HF2 on Tuesday [Dec 15] night, we believe the Orion Platform now meets the US Federal and state agencies’ requirements.”
But the trojanized releases were certified, weren’t they?
“We are continuing to take measures to ensure our internal systems are secure, including deploying the Falcon Endpoint Protection Platform across the endpoints on our systems.”
Falcon is CrowdStrike’s product. Surely it will name the nation-state, as they did in 2016.
“We have retained industry-leading third-party cybersecurity experts to assist us with this work and are actively collaborating with our partners, vendors, law enforcement and intelligence agencies around the world.”
They ARE cybersecurity experts themselves, and their product was shipped with the malware. They must preserve all the relevant data for the future legal proceedings, look through the old versions of the build code, find who and when inserted the malicious code, and surrender him or her to the FBI. This is, of course, if the person(s) responsible are not in China yet.
“The vulnerability was not evident in the Orion Platform products’ source code but appears to have been inserted during the Orion software build process.”
By insiders, although they did not say that.
(SWI 3) SolarWinds Security Statement, captured on Dec 9:
“SolarWinds maintains a written Information Security policy that defines employee’s responsibilities and acceptable use of information system resources.“
“Information security roles and responsibilities are defined within the organization.”
“SolarWinds follows the NIST Cybersecurity Framework with layered security controls to help identify, prevent, detect, and respond to security incidents.”
“SolarWinds maintains a change management process to ensure that all changes made to the production environment are applied in a deliberate manner.”
“SolarWinds does not give our suppliers or vendors direct access to network/equipment management responsibility.”
“We maintain audit logs on systems. These logs provide an account of which personnel have accessed which systems. Access to our auditing and logging tool is controlled by limiting access to authorized individuals. Security events are logged, monitored, and addressed by trained security team members.”
“SolarWinds maintains separate development and production environments.”
“Role based access controls are implemented for access to information systems. … Access controls to sensitive data in our databases, systems, and environments are set on a need-to-know / least privilege necessary basis.”
“We require that authorized users be provisioned with unique account IDs. Our password policy covers all applicable information systems, applications, and databases.”
“We follow a defined methodology for developing secure software that is designed to increase the resiliency and trustworthiness of our products.”
Being a network security company with such elaborate internal procedures, SWI would have known all the details behind the malware in its product very quickly, certainly within a week after being notified of the problem.
FireEye
(FEYE 1) FEYE filing with SEC, Dec 8
On December 8, 2020, concurrently with the filing of this Current Report on Form 8-K, FireEye, Inc. (“FireEye”, “we”, “our” or “us”) is announcing on our corporate blog that FireEye recently was attacked by a highly sophisticated cyber threat actor, one whose discipline, operational security, and techniques lead us to believe it was a state-sponsored attack. Based on his 25 years in cyber security and responding to incidents, Kevin Mandia, our Chief Executive Officer, concluded we are witnessing an attack by a nation with top-tier offensive capabilities.
Yes, this is the wording used by FireEye in their SEC report! FireEye reports to the SEC that it made a blog post. This is not a lie.
(FEYE 2) FireEye blog post by the CEO Kevin Mandia, Dec 8
“Based on my 25 years in cyber security and responding to incidents, I’ve concluded we are witnessing an attack by a nation with top-tier offensive capabilities. This attack is different from the tens of thousands of incidents we have responded to throughout the years. The attackers tailored their world-class capabilities specifically to target and attack FireEye.”
“We are actively investigating in coordination with the Federal Bureau of Investigation and other key partners, including Microsoft.”
“During our investigation to date, we have found that the attacker targeted and accessed certain Red Team assessment tools that we use to test our customers’ security. These tools mimic the behavior of many cyber threat actors and enable FireEye to provide essential diagnostic security services to our customers.”
(FEYE 3) FireEye blog post by FireEye, Dec 8
“A highly sophisticated state-sponsored adversary stole FireEye Red Team tools. Because we believe that an adversary possesses these tools, and we do not know whether the attacker intends to use the stolen tools themselves or publicly disclose them, FireEye is releasing hundreds of countermeasures with this blog pos…”
“A Red Team is a group of security professionals authorized and organized to mimic a potential adversary’s attack or exploitation capabilities against an enterprise’s security posture. … we have built up a set of scripts, tools, scanners, and techniques to help improve our clients’ security postures. Unfortunately, these tools were stolen by a highly sophisticated attacker.”
“Some of the tools are publicly available tools modified to evade basic security detection mechanisms. Other tools and frameworks were developed in-house for our Red Team.”
Why would a “highly sophisticated attacker” need their tools?
(FEYE 4) FireEye blog post, Dec 13
“This campaign may have begun as early as Spring 2020 and is currently ongoing.”
“SolarWinds.Orion.Core.BusinessLayer.dll is a SolarWinds digitally-signed component of the Orion software framework that contains a backdoor that communicates via HTTP to third party servers. We are tracking the trojanized version of this SolarWinds Orion plug-in as SUNBURST.”
Now the key evidence:
“The malware masquerades its network traffic as the Orion Improvement Program (OIP) protocol and stores reconnaissance results within legitimate plugin configuration files allowing it to blend in with legitimate SolarWinds activity.”
“The C2 traffic to the malicious domains is designed to mimic normal SolarWinds API communications.”
“Multiple trojanzied updates were digitally signed from March – May 2020 and posted to the SolarWinds updates website …”
The malware comes from SolarWinds, compiled into the SolarWinds software, uses SolarWinds network protocols, and signed with SolarWinds certificate. Guess who is the attacker behind the malware?
MSFT
(MSFT 1) Analyzing Solorigate, Dec 18
This article is the single most useful technical description of the SolarWinds malware.
“The fact that the compromised file is digitally signed suggests the attackers were able to access the company’s software development or distribution pipeline. Evidence suggests that as early as October 2019, these attackers have been testing their ability to insert code by adding empty classes. Therefore, insertion of malicious code into the SolarWinds.Orion.Core.BusinessLayer.dll likely occurred at an early stage, before the final stages of the software build, which would include digitally signing the compiled code. As a result, the DLL containing the malicious code is also digitally signed, which enhances its ability to run privileged actions—and keep a low profile.”
But even this article fails to mention that the SolarWinds malware infrastructure was hosted by Microsoft itself, as reported by DomainTools. After December 12, Microsoft seized the malware infrastructure through a court.
(MSFT 2) Customer Guidance on Recent Nation-State Cyber Attacks, Dec 13
“This post contains technical details about the methods of the actor we believe was involved in Recent Nation-State Cyber Attacks…”
(MSFT 3) Important steps for customers to protect themselves from recent nation-state cyberattacks, Dec 13
“Today, Microsoft is sharing information and issuing guidance about increased activities from a sophisticated threat actor that is focused on high value targets such as government agencies and cybersecurity companies. We believe this is nation-state activity at significant scale, aimed at both the government and private sector. “
These documents circularly refer to each other to justify the belief that the SolarWinds malware is a nation-state activity, and do not mention the nation-state. Neither of them mention that the attacker’s infrastructure was hosted by Microsoft at the time of the attack. The only non-circular reference is to FireEye.
(MSFT 4) A moment of reckoning, Brad Smith, Microsoft President, Dec 17
Microsoft President Brad Smith is a deranged Trump hater. This article refers to “incoming Biden-Harris Administration” four times.
“While investigations (and the attacks themselves) continue, Microsoft has identified and has been working this week to notify more than 40 customers that the attackers targeted more precisely and compromised through additional and sophisticated measures.”
There are many more
Russian engineers in 2016 identified weaknesses in password protection and social media platforms, hacked their way into American political campaigns, and used disinformation to sow divisions among the electorate.
Here, Brad Smith mixes up two parts of the Russia hoax. The reason for doing this is the Russian government’s demand that Microsoft should not blame Russia for any cyber incidents – and the treacherous Microsoft executives complied! No, it is not treason, but it does seem to violate 18 U.S. Code § 951. So they blame Russian engineers or territory. Microsoft’s and other Big Tech dependence on Russia is very small and yet they are still complying with a foreing government’s demands. I cannot even imagine what they are doing for the government of China, whose influence in the Tech world is incredibly vaster.
US Government
https://cyber.dhs.gov/ed/21-01/
https://us-cert.cisa.gov/ncas/alerts/aa20-352a
https://us-cert.cisa.gov/ncas/current-activity/2020/12/08/theft-fireeye-red-team-tools
Other
“Unnamed sources have told Reuters that they believe Russian hackers are behind the attack. Russia has denied any involvement. ‘I reject these statements, these accusations, once again,’ said Dmitry Peskov, a spokesperson for President Vladimir Putin, told Russia’s Tass news agency.
‘Even if it is true there have been some attacks over many months and the Americans managed to do nothing about them, possibly it is wrong to groundlessly blame Russians right away. We have nothing to do with this.’” (emphasis added)
We look like idiots.
Was it not astounding that just a while ago Klaus Schwab the founder and executive chairman of the World Economic Forum, stated: We all know, but still pay insufficient attention, to the frightening scenario of a comprehensive cyber attack could bring a complete halt to the power supply, transportation, hospital services, our society as a whole.
Is this just the begining?