www.mandiant.com
Open in
urlscan Pro
2606:4700:300b::a29f:f17d
Public Scan
URL:
https://www.mandiant.com/resources/blog/esxi-hypervisors-detection-hardening
Submission: On September 29 via api from US — Scanned from DE
Submission: On September 29 via api from US — Scanned from DE
Form analysis 2 forms found in the DOM
GET /search
<form action="/search" method="get">
<div class="js-form-item form-item js-form-type-textfield form-item-search js-form-item-search">
<label class="visually-hidden" for="edit-search">Search</label>
<input data-drupal-selector="edit-search" type="text" id="edit-search" name="search" value="" size="30" maxlength="128" class="form-text" placeholder="Search">
</div>
<div data-drupal-selector="edit-actions" class="form-actions js-form-wrapper form-wrapper" id="edit-actions">
<button data-drupal-selector="edit-submit-acquia-search" type="submit" id="edit-submit-acquia-search" class="button js-form-submit form-submit">
<span class="visually-hidden">Submit search form</span>
<svg width="16" height="17" viewBox="0 0 16 17" fill="none" xmlns="http://www.w3.org/2000/svg" aria-hidden="true">
<path d="M7.22574 13.9446C10.6622 13.9446 13.4481 11.1588 13.4481 7.72232C13.4481 4.28583 10.6622 1.5 7.22574 1.5C3.78925 1.5 1.00342 4.28583 1.00342 7.72232C1.00342 11.1588 3.78925 13.9446 7.22574 13.9446Z" stroke="currentColor"
stroke-width="2" stroke-linecap="round" stroke-linejoin="round"></path>
<path d="M15.0001 15.4996L11.6167 12.1162" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"></path>
</svg>
</button>
</div>
</form>GET /search
<form action="/search" method="get">
<div class="js-form-item form-item js-form-type-textfield form-item-search js-form-item-search">
<label class="visually-hidden" for="edit-search">Search</label>
<input data-drupal-selector="edit-search" type="text" id="edit-search" name="search" value="" size="30" maxlength="128" class="form-text" placeholder="Search">
</div>
<div data-drupal-selector="edit-actions" class="form-actions js-form-wrapper form-wrapper" id="edit-actions">
<button data-drupal-selector="edit-submit-acquia-search" type="submit" id="edit-submit-acquia-search" class="button js-form-submit form-submit">
<span class="visually-hidden">Submit search form</span>
<svg width="16" height="17" viewBox="0 0 16 17" fill="none" xmlns="http://www.w3.org/2000/svg" aria-hidden="true">
<path d="M7.22574 13.9446C10.6622 13.9446 13.4481 11.1588 13.4481 7.72232C13.4481 4.28583 10.6622 1.5 7.22574 1.5C3.78925 1.5 1.00342 4.28583 1.00342 7.72232C1.00342 11.1588 3.78925 13.9446 7.22574 13.9446Z" stroke="currentColor"
stroke-width="2" stroke-linecap="round" stroke-linejoin="round"></path>
<path d="M15.0001 15.4996L11.6167 12.1162" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"></path>
</svg>
</button>
</div>
</form>Text Content
Skip to main content
* Platform
MANDIANT ADVANTAGE
Multi-vendor XDR platform that delivers expertise and frontline intelligence
to security teams of all sizes.
Platform overview
PLATFORM MENU
* Automated Defense
Rapid event investigation & remediation
* Attack Surface Management
Map your external environment
* Security Validation
Validate controls are working properly
* Threat Intelligence
Integrate latest intel from the frontlines
* Ransomware Defense Validation
Test your ability to prevent ransomware
* Digital Threat Monitoring
Visibility into the open, deep and dark web
* Managed Defense
Eliminate threats with managed detection and response services
Get started for freeRegister for Mandiant Advantage Threat intelligence
* Solutions
MANDIANT SOLUTIONS
Solve your toughest cyber security challenges with use-case and
industry-focused combinations of our products and services.
SOLUTIONS MENU
* Use Cases
* Ransomware
Increase resilience against multifaceted extortion
* Cyber Threat Visibility
Know who is targeting you
* Cyber Risk Management
Advance your business approach to cyber security
* Attack Surface Visibility
See what attackers see
* Digital Risk Protection
Focus on what's most important to mitigate digital risk
* Cyber Preparedness
Validate your cyber preparedness
* OT/ICS Security
Extend cyber defense to strengthen OT and ICS security
* Detection and Response
Focus on what's most important to mitigate digital risk
* Insider Threats
Uncover and manage insider threats
* Cyber Security Skills Gap
Close gaps with flexible access to security experts
* Manufacturing
Manufacturing organizations know they must keep production lines
running.
* Public Sector
* Government Cyber Security
Protecting Governments from Cyber Attacks
* Election Security
Focus on Election Infrastructure Protection
* Intelligence
* Services
MANDIANT SERVICES
Mitigate threats, reduce risk and get back to business with the help of
experts!
Learn more
SERVICES MENU
* Featured Consulting Solutions
* Cyber Defense Transformation
Properly establish cyber defenses
* Incident Response
Tackle breaches confidently
* Strategic Readiness
Increase resilience to risk
* Technical Assurance
Test your security program
* View all Services (48)
* Expertise On Demand
Access to Mandiant Experts
* Training
* Find a Course
Browse on-demand and live training
* Mandiant Academy
Train your teams to protect effectively
Schedule a consultationGet in touch with a Mandiant expert
* Resource
RESOURCE
* Resource
* Mandiant Blog
Expert perspectives and industry news.
* Podcasts
Interviews, hot topics, and more
* Customer Stories
Case studies and customer testimonials.
* Reports
Research from the frontlines
* Webinars
Pre-recorded or livestreamed speaker events
* Insights
Cyber security concepts, methods, etc.
* Events
Conferences and collaborative events
* Infographics
Visualizations of security research and processes
* Datasheets
Descriptions of Mandiant offerings
newM-Trends 2022: Cyber Security Metrics, Insights and Guidance From the
FrontlinesLearn More
View all resources
* Company
COMPANY
Learn more about us and our mission to help organizations defend against
cyber crime.
About Mandiant
Contact Us
COMPANY MENU
* Careers
Life at Mandiant and open roles
* Noteholder Documents
* Media Center
News, reporting and research
* Partners
Partner ecosystem and resources
* Elevate
* Mandiant Gives Back
* Create a free account
* Sign in to Advantage
en expand_more
* English
* Français
* Deutsch
* Italian
* 日本
* 한국어
* Español
Start for Free
Search
Submit search form
Search
Submit search form
* Platform
* Mandiant Advantage Overview
* Automated Defense
* Security Validation
* Ransomware Defense Validation
* Attack Surface Management
* Threat Intelligence
* Digital Threat Monitoring
* Managed Defense
* Solutions
* Ransomware
* Cyber Risk Management
* Digital Risk Protection
* OT/ICS Security
* Insider Threats
* Cyber Security Skills Gap
* Manufacturing
* Election Security
* Government Cyber Security
* Cyber Threat Visibility
* Attack Surface Visibility
* Cyber Preparedness
* Detection and Response
* Intelligence
* Services
* Services Overview
* Incident Response
* Strategic Readiness
* Technical Assurance
* View all Services (48)
* Mandiant Academy
* Find a Course
* Expertise On Demand
* Resources
* Resources
* Mandiant Blogs
* Customer Stories
* Webinars
* Events
* Podcasts
* Reports
* Insights
* Datasheets
* Infographics
* Company
* About Mandiant
* Careers
* Media Center
* Partners
* Elevate
* Mandiant Gives Back
* Mobile Footer Section
* See what’s new at Mandiant
* Get started
* Incident Response Help
* Contact Sales
* Support
* Sign In
* Create a Free Mandiant Advantage Account
TOP
* Incident Response
* Contact sales
* Support
* Advantage Free Trial
* Blog
* Support
* Contact us
* report_problemIncident Response Assistance
BREADCRUMB
1. Home
2. Resources
3. Bad VIB(E)s Part Two: Detection and Hardening within ESXi Hypervisors
Blog
BAD VIB(E)S PART TWO: DETECTION AND HARDENING WITHIN ESXI HYPERVISORS
Alexander Marvi, Greg Blaum
Sep 29, 2022
9 mins read
Detection
In part one, we covered attackers’ usage of malicious vSphere Installation
Bundles (“VIBs”) to install multiple backdoors across ESXi hypervisors, focusing
on the malware present within the VIB payloads. In this installment, we will
continue to elaborate further on other attacker actions such as timestomping,
describe ESXi detection methodologies to dump process memory and perform YARA
scans, and discuss how to further harden hypervisors to minimize the attack
surface of ESXi hosts. For more details, VMware has released additional
information on protecting vSphere.
ESXI LOGGING
Both VIRTUALPITA and VIRTUALPIE stop the vmsyslogd process from recording
activity on startup, but multiple logging processes exist across the hypervisor
which can still be used to track attacker activity.
MALICIOUS VIB INSTALLATION
It was previously established in part one that ESXi systems do not allow for a
falsified VIB file below the minimal set acceptance level, even when the
acceptance level was modified in the Descriptor XML. To circumvent this, the
attacker abused the --force flag to install malicious CommunitySupported VIBs.
This flag adds a VIB or image profile with a lower acceptance level than
required by the host.
Evidence of the --force flags usage to install a VIB was found across multiple
locations on the ESXi hypervisor. The ESXi profile XML file records all VIBs
that have been installed on the system, specifying the date, time, and flags
used to install each VIB. This file is found under the path
/var/db/esximg/profile. Figure 1 contains an example of the attacker’s --force
flag usage logged in the profile XML file.
Figure 1: ESXI Profile XML file with the presence of a --force installation
The log file /var/log/esxupdate.log also recorded the usage of the --force flag
when a VIB is installed. Figure 2 contains an event that logged a malicious VIB
being installed with a forced installation.
Figure 2: VIB Installation with force flag in esxupdate.log
TIMESTOMPING
Mandiant observed that logs surrounding VIB installations with the --force flag
were recorded as early as October 9, 2013, which did not align with the attack
timeline. The log file /var/log/vmkwarning.log provided further evidence of the
system time being manipulated. Figure 3 contains two (2) events that logged the
system clock being modified right before and after attacker actions occurred.
This behavior suggests timestomping was being performed to cover up the true
time the attacker initially installed the VIBs on the machine.
Figure 3: vmkwarning.log recording system time modification
CREATION OF SYSCLOG
Analyzing the VIRTUALPITA sample rhttpproxy-io
(2c28ec2d541f555b2838099ca849f965), it was found that the sample listened over
the VMCI port number 18098. Once the listener is setup, the malware fetches the
system's CID (context ID) by issuing IOCTL request code 1977. The PID of the
backdoor, CID and listening port are then logged to /var/log/sysclog in the
following format [<date/timestamp>]\n\r[!]<<PID>>:<CID>:<port>\n\n as seen in
Figure 4.
Figure 4: Sample of sysclog
GUEST MACHINE INTERACTION
Further interactions between hypervisors and their respective guest machines
were discovered within multiple logs named vmware.log. These logs, located at
the following path /vmfs/volumes/…/<virtual machine hostname>/vmware.log, record
basic operations between the host and hypervisor that were not logged on the
endpoint. Actions recorded by this log include guest machine logins,
file/directory creation and deletion, command execution, and file transfer
between guest machine and hypervisor. To focus on interactions between the
hypervisor and its guest machines in the vmware.log, filter for lines containing
GuestOps.
VIB VERIFICATION AT SCALE
The previous blog post touched on using the command esxcli software vib
signature verify to identify any VIBs that do not pass the signature
verification check made by the ESXi hypervisor. Alternative VIB configurations
exist that would be able to circumvent the signature verification check.
Mandiant confirmed that when a VIB is installed as CommunitySupported, the
Signature Verification field will label it as Succeeded if the payload is not
tampered with after installation. This means a VIB could be created without any
validation from VMWare or its partners and still be labelled as validated.
To account for properly signed CommunitySupported VIBs and other anomalous
configurations which could indicate malicious activity, all VIBs in the
environment can be compared with a list of known good VIBs. A matrix created by
VMware Front Experience breaks down the names and builds of each VIB expected to
be present by default in the respective ESXi build. Each time a VIB is changed
across ESXi builds the matrix links to the official VMware patch release notes
which state the adding, modification, or removal of that VIB. A sample of this
matrix can be seen in Figure 5.
Figure 5: Sample of Known Good VIB Matrix
ESXI DETECTION METHODOLOGIES
While ESXi shares many similarities to Linux (commands, directory structure,
etc.), it is entirely its own operating system known as VMkernel, meaning
popular methods to scan the filesystem and dump process memory do not work.
Mandiant has formulated alternative detections methods to attempt to provide
investigators with better visibility into ESXi hypervisors during future
incidents.
REMOTE ESXI YARA SCANNING WITH SSHFS
Multiple YARA rules were generated for the detection of VIRTUALPITA and
VIRTUALPIE across Linux and ESXi environments and can be found in the first part
of this blog post. These detections have two caveats to them based on the
storage and execution of the malware. If the attacker is launching either
malware family from a VIB on ESXi, the sample within the VIB will not be
detected due to being compressed in the .vgz format. Secondly, if the binary is
running in memory but deleted from disk, the binary will not be detected by
YARA’s file system scan.
Since YARA does not run directly on ESXi hosts, Mandiant utilized sshfs to
perform remote YARA scanning of ESXi hypervisors.
PREREQUISITES
Note: All behaviors of ESXi and the methodology to dump memory have been
confirmed for ESXi 6.7, no other versions at this time have been tested.
Before scanning the ESXi machine a few prerequisites must be met. For the ESXi
machine which the memory is being dumped, you must have both:
* Root Access to the machine
* SSH Enabled on the ESXi Server
Once the ESXi machine is correctly configured, a Linux machine must be setup to
be able to communicate over SSH with the ESXi machine. This Linux machine must
also install:
* sshfs
* yara
PERFORMING THE YARA SCAN
Note: Since YARA will naturally recursively scan directories and sshfs pulls
files back as they are accessed, scanning the entire ESXi file system can take a
long time depending on network bandwidth. This method of scanning is only
suggested if a strong and stable network connection is present.
Linux Commands
Description
Commands
Create a directory to mount the ESXi machine on
> mkdir /mnt/yara
Mount the ESXi root directory to the Linux machine mount point using sshfs
> sshfs -o allow_other,default_permissions root@<Insert ESXi IP Address>:/
/mnt/yara
Scan the mount point which the ESXi system is attached to
> yara -r <Provided YARA Rule> <scope of scan>
DUMPING ESXI PROCESS MEMORY
When attempting to dump the process memory from a ESXi hypervisor like you would
a Linux machine, it will quickly become apparent that the /proc/ directory will
be either empty or containing a single PID of the commands used to attempt to
dump the memory. To recover process memory from ESXi (and potentially the full
binary itself), a mixture of the native tool gdbserver and a github tool called
core2ELF64 can be utilized.
PREREQUISITES
Note: All behaviors of ESXi and the methodology to dump memory have been
confirmed for ESXi 6.7, no other versions at this time have been tested.
Before dumping the process memory a few prerequisites must be met. For the ESXi
machine which, you must have both:
* Root Access to the machine
* SSH Enabled on the ESXi Server
Once the ESXi machine is correctly configured, a Linux machine must be setup to
be able to communicate over SSH with the ESXi machine. This Linux machine must
also install:
* gdb
* core2ELF64
DUMPING MEMORY
Note: The ports to listen and port forward through are arbitrary (Rule of Thumb:
Keep between 1024-25565 to avoid commonly used ports), for this walkthrough the
listening port will be 6000 and the forwarding port will be 7000.
To dump the ESXi process memory, gdbserver will be utilized to hook into the
currently running process, specified by PID, and listen on an arbitrary port.
ESXi Commands
Description
Commands
A preemptive check used to make sure that the PID you will be collecting in the
next command is the intended one. Please make sure that the output of this
statement only shows the process you intend to dump the memory for.
> ps -Tcjstv | grep -e “<Binary to Dump>”
Attaches gdbserver to the PID specified in the list processes command, listening
on port 6000 for gdb to connect to.
> gdbserver –attach 127.0.0.1:6000 `ps -Tcjstv |
grep -e “<Binary to Dump>” | awk ‘{print $1}’ | head -n 1`
Once listening, the Linux machine will create an SSH tunnel (Port Forward) to
the listening port on the ESXi server, where gdb will be used to create a core
dump of the process specified.
Linux Commands
Description
Commands
Set up an SSH tunnel from the Linux machine to the listening port of the ESXi
Server gdbserver process.
> ssh -L 1336:127.0.0.1:6000 -f -N <acct on ESX>@<IP of ESX>
Launch gdb
> gdb
Within the gdb shell, connect to the gdbserver instance. If at any point you
have successfully ran this command and leave the gdb shell, you will need to
exit and relaunch the gdbserver process on ESXi to reconnect.
(gdb) > target remote localhost:1336
Create a core dump file of the attach processes' memory in the working
directory. The output file should be the following syntax "core.[0-9]{7}".
?? () > gcore
PROCESS EXTRACTION
Once a core file is created, the Github project core2ELF64 can be used to
reconstruct the program.
Linux Commands
Description
Commands
Set up an SSH tunnel from the Linux machine to the listening port of the ESXi
Server gdbserver process.
In the event of the program not being able to recover the first segment, choose
the next available segment possible (Smallest Number)
> core2ELF64 <core file> <Desired Output Name>
SOURCES
* Hooking into ESXi processes with gdbserver
HARDENING ESXI
NETWORK ISOLATION
When configuring networking on the ESXi hosts, only enable VMkernel network
adapters on the isolated management network. VMkernel network adapters provide
network connectivity for the ESXi hosts and handle necessary system traffic for
functionality such as vSphere vMotion, vSAN and vSphere replication. Ensure that
all dependent technologies such as vSANs and backup systems that the
virtualization infrastructure will use are available on this isolated network.
If possible, use dedicated management systems exclusively connected to this
isolated network to conduct all management tasks of the virtualization
infrastructure.
IDENTITY AND ACCESS MANAGEMENT
Consider decoupling ESXi and vCenter Servers from Active Directory and use
vCenter Single Sign-On. Removing ESXi and vCenter from Active Directory will
prevent any compromised Active Directory accounts from being able to be used to
authenticate directly to the virtualization infrastructure. Ensure
administrators use separate and dedicated accounts for managing and accessing
the virtualized infrastructure. Enforce multi-factor authentication (MFA) for
all management access to vCenter Server instances and store all administrative
credentials in a Privileged Access Management (PAM) system.
SERVICES MANAGEMENT
To further restrict services and management of ESXi hosts, implement lockdown
mode. This ensures that ESXi hosts can only be accessed through a vCenter
Server, disables some services, and restricts some services to certain defined
users. Configure the built-in ESXi host firewall to restrict management access
only from specific IP addresses or subnets that correlate to management systems
on the isolated network. Determine the appropriate risk acceptance level for
vSphere Installable Bundles (VIBs) and enforce acceptance levels in the Security
Profiles for ESXi hosts. This protects the integrity of the hosts and ensures
unsigned VIBs cannot be installed.
LOG MANAGEMENT
Centralized logging of ESXi environments is critical, both to proactively detect
potential malicious behavior and investigate an actual incident. Ensure all ESXi
host and vCenter Server logs are being forwarded to the organization’s SIEM
solution. This provides visibility into security events beyond that of normal
administrative activity.
ACKNOWLEDGEMENTS
Special thanks to Brad Slaybaugh, Joshua Kim, Zachary Smith, Jeremy Koppen, and
Charles Carmakal for their assistance with the investigation, technical review,
and creating detections/investigative methodologies for the malware families
discussed in this blog post. In addition, we would also like to thank VMware
their collaboration on this research.
* Follow us
*
*
*
*
FOOTER
* Mandiant Advantage Platform
* Platform Overview
* Automated Defense
* Security Validation
* Ransomware Defense Validation
* Attack Surface Management
* Threat Intelligence
* Digital Threat Monitoring
* Managed Defense
* Solutions
* Ransomware
* Industrial Controls & OT
* Cyber Risk Management
* Digital Risk Protection
* Insider Threats
* Cyber Security Skills Gap
* Election Security
* Government Cyber Security
* Manufacturing
* Cyber Threat Visibility
* Attack Surface Visibility
* Cyber Preparedness
* Detection and Response
* Services
* Services Overview
* Incident Response
* Strategic Readiness
* Cyber Defense Transformation
* Technical Assurance
* View all Services (48)
* Expertise on Demand
* Mandiant Academy
* Overview
* Education Formats
* Upcoming Courses
* On-Demand Courses
* Certifications
* ThreatSpace Cyber Range
* Free Course Sneak Peaks
* Resources
* Resource Center
* Mandiant Blog
* Podcasts
* Customer Stories
* Reports
* Webinars
* Insights
* Infographics
* Datasheets
* Company
* About Us
* Careers
* Events
* Media Center
* Noteholder Documents
* Partners
* Partners Overview
* Service Partners
* Cyber Risk Partners
* Technology Partners
* Partner Portal
* Connect with Mandiant
* Contact Us
* Report an Incident
* Customer Support
* Email Preferences
* Customer Success
* Media Inquiries
© Copyright 2022 Mandiant. All rights reserved.
BOTTOM
* Privacy & Cookies Policy
* Terms & Conditions
* Compliance
* Site Map
By clicking “Accept All Cookies”, you agree to the storing of cookies on your
device to enhance site navigation, analyze site usage, and assist in our
marketing efforts. Privacy Policy
Cookies Settings Reject All Accept All Cookies
PRIVACY PREFERENCE CENTER
When you visit any website, it may store or retrieve information on your
browser, mostly in the form of cookies. This information might be about you,
your preferences or your device and is mostly used to make the site work as you
expect it to. The information does not usually directly identify you, but it can
give you a more personalized web experience. Because we respect your right to
privacy, you can choose not to allow some types of cookies. Click on the
different category headings to find out more and change our default settings.
However, blocking some types of cookies may impact your experience of the site
and the services we are able to offer.
More information
Allow All
MANAGE CONSENT PREFERENCES
REQUIRED COOKIES
Always Active
These cookies are necessary for the website to function and cannot be switched
off in our systems. They are usually only set in response to actions made by you
which amount to a request for services, such as setting your privacy
preferences, logging in or filling in forms. You can set your browser to block
or alert you about these cookies, but some parts of the site will not then work.
These cookies do not store any personally identifiable information.
PERFORMANCE COOKIES
Performance Cookies
These cookies allow us to count visits and traffic sources so we can measure and
improve the performance of our site. They help us to know which pages are the
most and least popular and see how visitors move around the site. All
information these cookies collect is aggregated and therefore anonymous. If you
do not allow these cookies we will not know when you have visited our site, and
will not be able to monitor its performance.
TARGETING COOKIES
Targeting Cookies
These cookies may be set through our site by our advertising partners. They may
be used by those companies to build a profile of your interests and show you
relevant adverts on other sites. They do not store directly personal
information, but are based on uniquely identifying your browser and internet
device. If you do not allow these cookies, you will experience less targeted
advertising.
Back Button
PERFORMANCE COOKIES
Search Icon
Filter Icon
Clear
checkbox label label
Apply Cancel
Consent Leg.Interest
checkbox label label
checkbox label label
checkbox label label
Reject All Confirm My Choices