Troubleshooting Anti-Malware Engine Offline errors in Deep Security

Make sure that the vShield Endpoint Thin Agent driver is installed on the virtual machine. To check if it is running:

1. Open the run dialog box in the virtual machine.
2. Type the "msinfo32" command.
3. Go to Software Environment > System Drivers.
4. Make sure that the VM drivers, vmci, and vsepflt, are running.

   

By default, VMware tools installation do not install vShield drivers. Perform installation using the Complete or Custom option and manually select the Guest Introspection drivers.

Verify that the Deep Security Manager is syncing properly with the vCenter Server and NSX Manager.

1. On the DSM console, go to Computers.
2. Right-click the vCenter Server and choose Properties.
3. On the General tab, click Synchronize Now.

   

4. If the sync fails, check if the vCenter Server and NSX Manager credentials are correct. The vCenter/NSX Manager certificate may also need to be updated.

Check if the vCenter Server is properly registered and connected to the NSX Management Service.

For the complete procedure, refer to this VMware article: Register vCenter Server with NSX Manager.

Verify that the Guest Introspection and Deep Security service deployments has no errors under Networking and Security > Installation > Service Deployments tab.

Before Deep Security Manager 11.0, when deploying DSVA from NSX manager with NSX for vShield Endpoint license, the "VMware Network Fabric" missing dependency alert will pop up. It is suggested to ignore the alert and force the DSVA deployment by clicking Failed and then click Resolve. The DSVA deployment will be successful, but the status of Trend Micro Deep Security still shows "Failed". You can ignore it as no Deep Security function actually fails.

Make sure that NSX Security Groups and NSX Policy for Deep Security exist and the affected VM exist on the Security Group's VM list.

1. In vSphere Web Client, go to Home > Networking & Security > Service Composer > Security Groupsand make sure that Deep Security group exist.

   

2. Go to Home > Networking & Security > Service Composer > Security Policies and make sure that Deep Security policy exist.

   

   If either of Security Group or Security Policy for Deep Security does not exist, create these by following the instructions in an article from the Deep Security Help Center: Create NSX security groups and policies.
3. Go to Home > Networking & Security > Service Composer > Security Groupsand make sure that the affected VM exists in list of VMs on the Deep Security Group.

   

   If the affected VM doesn't exist on this list, edit the Deep Security Group and modify the "Select objects to include" so that it includes the cluster where the agentless protected VM resides. Refer to the Deep Security Help Center article: Create NSX security group and policies.

On the Deep Security Manager console, make sure that the Trend Micro Deep Security Appliance version is displayed as higher than 9.5.2-2202. The appliance's initial version is 9.5.2-2202 which will be automatically upgraded to higher version provided that the Deep Security Manager has the latest Agent-RedHat_EL6 package available in it's software list.

Check the Deep Security Virtual Appliance (DSVA).

1. Log on to DSVA via console or SSH. Follow the procedure in this article: Enabling SSH access on Deep Security Virtual Appliance (DSVA).
2. Execute the "ifconfig –a" command. The following should appear:

   

3. Check if the AM process listens to 48651.

   ~$netstat -ant | grep 48651

   

Check the vShield driver on ESXi.

1. Log on to ESX via SSH.
2. Execute the "ps | grep vShield-Endpoint-Mux" command. The following appears:

   

3. Verify if the vShield Endpoint Service on ESX/ESXi is running using the following command:

   ~ # /etc/init.d/vShield-Endpoint-Mux status

   If you need to restart, run the following:

   ~ # /etc/init.d/vShield-Endpoint-Mux restart

   

4. Verify that the DSVA is registered with the vShield Endpoint, run the following command:

   ~ # esxcfg-advcfg --get /UserVars/RmqIpAddress

   ~ # esxcfg-advcfg --get /UserVars/RmqPort

   

   Based on the sample above, RmpIpAddress is the NSX Manager.

5. Verify that the TCP connection between the vShield-Endpoint-Mux and DSVA is established:

   ~# esxcli network ip connection list | grep vShield

   

If it fails to connect to 169.254.1.1.24:48655, the following is visible on the ESXi syslog.log:

014-07-18T08:05:35Z EPSecMux[35465]: [INFO] (EPSEC) [0x8a89]  SolutionHandler[0x1f0026a0] connected to solution[100] at [169.254.1.24:48655]  2014-07-18T08:05:35Z EPSecMux[35465]: [WARNING] (EPSEC) [0x8a89] Unknown attribute: 0x19

As a workaround:

1. Open the Security Group on the NSX setting.
2. Exclude the offline guest VM.
3. Re-apply the policy to the Security Group.
4. Add the offline guest VM.
5. Re-apply the policy to the Security Group.

Transferring virtual machines through vMotion in NSX environment causes Anti-Malware Engine to show offline status. This happens because Deep Security Manager (DSM) has already checked the status of the VMs during heartbeat before the actual completion of vMotion process.

Waiting for the next heartbeat, which is 10 minutes by default, will fix the offline status.

As a workaround:

1. Right-click the affected virtual machine.
2. Select Actions and click Clear Warnings/Errors.
3. Click Actions again and click Check Status.

A virtual machine (VM) may also report an "Anti-Malware Engine Offline" status in Deep Security Manager (DSM) after executing a Storage vMotion in an activated state. Storage vMotion provides a new UUID to the virtual machine, as explained in this VMware article: Changing or keeping a UUID for a moved virtual machine.

The new UUID would be unknown to the vShield Manager/NSX, Deep Security Virtual Appliance, or DSM, but would be recognized as protected via vCenter connection. Therefore, DSM reports "Anti-Malware Engine Offline".

To prevent this situation, do the following:

1. Deactivate the VM before performing a Storage vMotion.
2. Perform the Storage vMotion.
3. Activate the VM again after the process.

You see the message "Anti-Malware Engine Offline" for the vApp VMs after they are deployed from a catalog/template from vCloud Director and integrated with Deep Security. The issue occurs because all instances and virtual machines deployed from a given catalog/vApp template from vCloud Director are given the same BIOS UUID.

Because Deep Security distinguishes different VMs by their BIOS UUID, a duplicate value in the vCenter causes an "Anti-Malware Engine Offline" error to the VMs in vCloud, when integrated with Deep Security.

To resolve the issue, create a new unique uuid.bios entry. For the procedure, refer to this VMware KB article: BIOS UUIDs in vCloud Director are not unique when virtual machines are deployed from catalog templates.