After activating agentless protection, a virtual machine (VM) may go from "Managed (Online)" to "Anti-Malware Engine Offline". This article consolidates the components that need to be checked as well as the other factors that may affect the functionality.
Before going further, it is recommended to identify first how many VMs are affected and if they are running on the same or different ESXi host(s). This can help isolate if the issue is on endpoint, host, cluster, or vCenter level. The following must also be checked:
- VMware and Deep Security compatibility. The vCenter, ESXi, and NSX Manager version should be supported. See compatibility matrix to verify.
- Relay Group. There should be at least one (1) functional member of the relay group. It can be checked from the Deep Security Manager (DSM) console and then go to Administration > Relay Management > Relay Group.
Make sure that the vShield Endpoint Thin Agent driver is installed on the virtual machine. To check if it is running:
- Open the run dialog box in the virtual machine.
- Type the "msinfo32" command.
- Go to Software Environment > System Drivers.
- Make sure that the VM drivers, vmci, and vsepflt, are running.
By default, VMware tools installation do not install vShield drivers. Perform installation using the Complete or Custom option and manually select the Guest Introspection drivers.
Verify that the Deep Security Manager is syncing properly with the vCenter Server and NSX Manager.
- On the DSM console, go to Computers.
- Right-click the vCenter Server and choose Properties.
- On the General tab, click Synchronize Now.
- If the sync fails, check if the vCenter Server and NSX Manager credentials are correct. The vCenter/NSX Manager certificate may also need to be updated.
- If the vCenter Synchronization is unusually slow or taking a significant amount of time (with no actual errors shown in DSM console), this could indicate a problem with synchronization. More likely, both the status of DSVA and Protected Agentless machines are not synchronized with DSM, thus the machines are shows as "Anti-Malware Engine Offline". In this case, check the server0.log of DSM, or try restart DSM Service and check if this helps solving the vCenter Synchronization issue. After a successful vCenter Synchronization, perform both a Clear Warnings/Errors and a Check Status on the affected agentless machines. This scenario has been observed to happen after DSM has been upgraded. The solution is to restart DSM Service or reboot the DSM Server.
Check if the vCenter Server is properly registered and connected to the NSX Management Service.
For the complete procedure, refer to this VMware article: Register vCenter Server with NSX Manager.
Verify that the Guest Introspection and Deep Security service deployments has no errors under Networking and Security > Installation > Service Deployments tab.
Before Deep Security Manager 11.0, when deploying DSVA from NSX manager with NSX for vShield Endpoint license, the "VMware Network Fabric" missing dependency alert will pop up. It is suggested to ignore the alert and force the DSVA deployment by clicking Failed and then click Resolve. The DSVA deployment will be successful, but the status of Trend Micro Deep Security still shows "Failed". You can ignore it as no Deep Security function actually fails.
Make sure that NSX Security Groups and NSX Policy for Deep Security exist and the affected VM exist on the Security Group's VM list.
- In vSphere Web Client, go to Home > Networking & Security > Service Composer > Security Groupsand make sure that Deep Security group exist.
Go to Home > Networking & Security > Service Composer > Security Policies and make sure that Deep Security policy exist.If either of Security Group or Security Policy for Deep Security does not exist, create these by following the instructions in an article from the Deep Security Help Center: Create NSX security groups and policies.
- Go to Home > Networking & Security > Service Composer > Security Groupsand make sure that the affected VM exists in list of VMs on the Deep Security Group.
If the affected VM doesn't exist on this list, edit the Deep Security Group and modify the "Select objects to include" so that it includes the cluster where the agentless protected VM resides. Refer to the Deep Security Help Center article: Create NSX security group and policies.
On the Deep Security Manager console, make sure that the Trend Micro Deep Security Appliance version is displayed as higher than 9.5.2-2202. The appliance's initial version is 9.5.2-2202 which will be automatically upgraded to higher version provided that the Deep Security Manager has the latest Agent-RedHat_EL6 package available in it's software list.
Check the Deep Security Virtual Appliance (DSVA).
- Log on to DSVA via console or SSH. Follow the procedure in this article: Enabling SSH access on Deep Security Virtual Appliance (DSVA).
- Execute the "ifconfig –a" command. The following should appear:
- Check if the AM process listens to 48651.
~$netstat -ant | grep 48651
Check the vShield driver on ESXi.
- Log on to ESX via SSH.
- Execute the "ps | grep vShield-Endpoint-Mux" command. The following appears:
- Verify if the vShield Endpoint Service on ESX/ESXi is running using the following command:
~ # /etc/init.d/vShield-Endpoint-Mux status
If you need to restart, run the following:
~ # /etc/init.d/vShield-Endpoint-Mux restart
- Verify that the DSVA is registered with the vShield Endpoint, run the following command:
~ # esxcfg-advcfg --get /UserVars/RmqIpAddress
~ # esxcfg-advcfg --get /UserVars/RmqPort
Based on the sample above, RmpIpAddress is the NSX Manager.
- Verify that the TCP connection between the vShield-Endpoint-Mux and DSVA is established:
~# esxcli network ip connection list | grep vShield
If it fails to connect to 169.254.1.1.24:48655, the following is visible on the ESXi syslog.log:
014-07-18T08:05:35Z EPSecMux: [INFO] (EPSEC) [0x8a89] SolutionHandler[0x1f0026a0] connected to solution at [169.254.1.24:48655] 2014-07-18T08:05:35Z EPSecMux: [WARNING] (EPSEC) [0x8a89] Unknown attribute: 0x19
As a workaround:
- Open the Security Group on the NSX setting.
- Exclude the offline guest VM.
- Re-apply the policy to the Security Group.
- Add the offline guest VM.
- Re-apply the policy to the Security Group.
Transferring virtual machines through vMotion in NSX environment causes Anti-Malware Engine to show offline status. This happens because Deep Security Manager (DSM) has already checked the status of the VMs during heartbeat before the actual completion of vMotion process.
Waiting for the next heartbeat, which is 10 minutes by default, will fix the offline status.
As a workaround:
- Right-click the affected virtual machine.
- Select Actions and click Clear Warnings/Errors.
- Click Actions again and click Check Status.
A virtual machine (VM) may also report an "Anti-Malware Engine Offline" status in Deep Security Manager (DSM) after executing a Storage vMotion in an activated state. Storage vMotion provides a new UUID to the virtual machine, as explained in this VMware article: Changing or keeping a UUID for a moved virtual machine.
The new UUID would be unknown to the vShield Manager/NSX, Deep Security Virtual Appliance, or DSM, but would be recognized as protected via vCenter connection. Therefore, DSM reports "Anti-Malware Engine Offline".
To prevent this situation, do the following:
- Deactivate the VM before performing a Storage vMotion.
- Perform the Storage vMotion.
- Activate the VM again after the process.
You see the message "Anti-Malware Engine Offline" for the vApp VMs after they are deployed from a catalog/template from vCloud Director and integrated with Deep Security. The issue occurs because all instances and virtual machines deployed from a given catalog/vApp template from vCloud Director are given the same BIOS UUID.
Because Deep Security distinguishes different VMs by their BIOS UUID, a duplicate value in the vCenter causes an "Anti-Malware Engine Offline" error to the VMs in vCloud, when integrated with Deep Security.
To resolve the issue, create a new unique uuid.bios entry. For the procedure, refer to this VMware KB article: BIOS UUIDs in vCloud Director are not unique when virtual machines are deployed from catalog templates.
If the issue persists, collect the following and send to Trend Micro Technical Support: