Sign In with your
Trend Micro Account
Need Help?
Need More Help?

Create a technical support case if you need further support.

Troubleshooting Anti-Malware Engine Offline errors in Deep Security

    • Updated:
    • 24 Oct 2018
    • Product/Version:
    • Deep Security 10.0
    • Deep Security 10.1
    • Deep Security 10.2
    • Deep Security 10.3
    • Deep Security 11.0
    • Deep Security 11.1
    • Deep Security 11.2
    • Deep Security 9.6
    • Platform:
    • VMware ESXi 5.1
    • VMware ESXi 6.0
    • VMware ESXi 6.5
    • VMware ESXi 6.7
Summary

After activating agentless protection, a virtual machine (VM) may go from "Managed (Online)" to "Anti-Malware Engine Offline". This article consolidates the components that need to be checked as well as the other factors that may affect the functionality.

Before going further, it is recommended to identify first how many VMs are affected and if they are running on the same or different ESXi host(s). This can help isolate if the issue is on endpoint, host, cluster, or vCenter level. The following must also be checked:

  • VMware and Deep Security compatibility. The vCenter, ESXi, and NSX Manager version should be supported. See compatibility matrix to verify.
  • Relay Group. There should be at least one (1) functional member of the relay group. It can be checked from the Deep Security Manager (DSM) console and then go to Administration > Relay Management > Relay Group.
Details
Public

Make sure that the vShield Endpoint Thin Agent driver is installed on the virtual machine. To check if it is running:

  1. Open the run dialog box in the virtual machine.
  2. Type the "msinfo32" command.
  3. Go to Software Environment > System Drivers.
  4. Make sure that the VM drivers, vmci, and vsepflt, are running.

    Check the VM drivers, vmci, and vsepflt are running

By default, VMware tools installation do not install vShield drivers. Perform installation using the Complete or Custom option and manually select the Guest Introspection drivers.

Manually select the Guest Introspection drivers

 
Depending on the VMware tools version, the Guest Introspection Drivers may be listed as vShield Drivers or NSX File Introspection Drivers.
 
Applies only to Deep Security On-Premise using VMware NSX

Verify that the Deep Security Manager is syncing properly with the vCenter Server and NSX Manager.

  1. On the DSM console, go to Computers.
  2. Right-click the vCenter Server and choose Properties.
  3. On the General tab, click Synchronize Now.

    Click Synchronize Now

  4. If the sync fails, check if the vCenter Server and NSX Manager credentials are correct. The vCenter/NSX Manager certificate may also need to be updated.
 
Applies only to Deep Security On-Premise using VMware NSX

Check if the vCenter Server is properly registered and connected to the NSX Management Service.

Check if the vCenter Server is properly registered and connected to the NSX Management Service

For the complete procedure, refer to this VMware article: Register vCenter Server with NSX Manager.

 
Applies only to Deep Security On-Premise using VMware NSX

Verify that the Guest Introspection and Deep Security service deployments has no errors under Networking and Security > Installation > Service Deployments tab.

Check the Guest Introspection and Deep Security under Service Deployments tab

Before Deep Security Manager 11.0, when deploying DSVA from NSX manager with NSX for vShield Endpoint license, the "VMware Network Fabric" missing dependency alert will pop up. It is suggested to ignore the alert and force the DSVA deployment by clicking Failed and then click Resolve. The DSVA deployment will be successful, but the status of Trend Micro Deep Security still shows "Failed". You can ignore it as no Deep Security function actually fails.

 
Applies only to Deep Security On-Premise using VMware NSX

Make sure that NSX Security Groups and NSX Policy for Deep Security exist and the affected VM exist on the Security Group's VM list.

  1. In vSphere Web Client, go to Home > Networking & Security > Service Composer > Security Groupsand make sure that Deep Security group exist.

    Check for Deep Security group

  2. Go to Home > Networking & Security > Service Composer > Security Policies and make sure that Deep Security policy exist.

    Check for Deep Security policy

    If either of Security Group or Security Policy for Deep Security does not exist, create these by following the instructions in an article from the Deep Security Help Center: Create NSX security groups and policies.
  3. Go to Home > Networking & Security > Service Composer > Security Groupsand make sure that the affected VM exists in list of VMs on the Deep Security Group.

    Check for affected VM

    If the affected VM doesn't exist on this list, edit the Deep Security Group and modify the "Select objects to include" so that it includes the cluster where the agentless protected VM resides. Refer to the Deep Security Help Center article: Create NSX security group and policies.

On the Deep Security Manager console, make sure that the Trend Micro Deep Security Appliance version is displayed as higher than 9.5.2-2202. The appliance's initial version is 9.5.2-2202 which will be automatically upgraded to higher version provided that the Deep Security Manager has the latest Agent-RedHat_EL6 package available in it's software list.

Check for DSA version

Check the Deep Security Virtual Appliance (DSVA).

  1. Log on to DSVA via console or SSH. Follow the procedure in this article: Enabling SSH access on Deep Security Virtual Appliance (DSVA).
  2. Execute the "ifconfig –a" command. The following should appear:

    Execute the ifconfig –a command

  3. Check if the AM process listens to 48651.

    ~$netstat -ant | grep 48651

    Check if the AM process listens to 48651.

Check the vShield driver on ESXi.

  1. Log on to ESX via SSH.
  2. Execute the "ps | grep vShield-Endpoint-Mux" command. The following appears:

    Execute ps | grep vShield-Endpoint-Mux command.

  3. Verify if the vShield Endpoint Service on ESX/ESXi is running using the following command:

    ~ # /etc/init.d/vShield-Endpoint-Mux status

    If you need to restart, run the following:

    ~ # /etc/init.d/vShield-Endpoint-Mux restart

    Restart vShield Endpoint Service

  4. Verify that the DSVA is registered with the vShield Endpoint, run the following command:

    ~ # esxcfg-advcfg --get /UserVars/RmqIpAddress

    ~ # esxcfg-advcfg --get /UserVars/RmqPort

    Verify that the DSVA is registered with the vShield Endpoint

    Based on the sample above, RmpIpAddress is the NSX Manager.

  5. Verify that the TCP connection between the vShield-Endpoint-Mux and DSVA is established:

    ~# esxcli network ip connection list | grep vShield

    Verify that the TCP connection between the vShield-Endpoint-Mux and DSVA is established

If it fails to connect to 169.254.1.1.24:48655, the following is visible on the ESXi syslog.log:

014-07-18T08:05:35Z EPSecMux[35465]: [INFO] (EPSEC) [0x8a89]  SolutionHandler[0x1f0026a0] connected to solution[100] at [169.254.1.24:48655]  2014-07-18T08:05:35Z EPSecMux[35465]: [WARNING] (EPSEC) [0x8a89] Unknown attribute: 0x19

As a workaround:

  1. Open the Security Group on the NSX setting.
  2. Exclude the offline guest VM.
  3. Re-apply the policy to the Security Group.
  4. Add the offline guest VM.
  5. Re-apply the policy to the Security Group.

Transferring virtual machines through vMotion in NSX environment causes Anti-Malware Engine to show offline status. This happens because Deep Security Manager (DSM) has already checked the status of the VMs during heartbeat before the actual completion of vMotion process.

Waiting for the next heartbeat, which is 10 minutes by default, will fix the offline status.

As a workaround:

  1. Right-click the affected virtual machine.
  2. Select Actions and click Clear Warnings/Errors.
  3. Click Actions again and click Check Status.

A virtual machine (VM) may also report an "Anti-Malware Engine Offline" status in Deep Security Manager (DSM) after executing a Storage vMotion in an activated state. Storage vMotion provides a new UUID to the virtual machine, as explained in this VMware article: Changing or keeping a UUID for a moved virtual machine.

The new UUID would be unknown to the vShield Manager/NSX, Deep Security Virtual Appliance, or DSM, but would be recognized as protected via vCenter connection. Therefore, DSM reports "Anti-Malware Engine Offline".

To prevent this situation, do the following:

  1. Deactivate the VM before performing a Storage vMotion.
  2. Perform the Storage vMotion.
  3. Activate the VM again after the process.

You see the message "Anti-Malware Engine Offline" for the vApp VMs after they are deployed from a catalog/template from vCloud Director and integrated with Deep Security. The issue occurs because all instances and virtual machines deployed from a given catalog/vApp template from vCloud Director are given the same BIOS UUID.

Because Deep Security distinguishes different VMs by their BIOS UUID, a duplicate value in the vCenter causes an "Anti-Malware Engine Offline" error to the VMs in vCloud, when integrated with Deep Security.

To resolve the issue, create a new unique uuid.bios entry. For the procedure, refer to this VMware KB article: BIOS UUIDs in vCloud Director are not unique when virtual machines are deployed from catalog templates.

If the issue persists, collect the following and send to Trend Micro Technical Support:

Premium
Internal
Rating:
Category:
Troubleshoot
Solution Id:
1104709
Feedback
Did this article help you?

Thank you for your feedback!

To help us improve the quality of this article, please leave your email here so we can clarify further your feedback, if neccessary:
We will not send you spam or share your email address.

*This form is automated system. General questions, technical, sales, and product-related issues submitted through this form will not be answered.

If you need additional help, you may try to contact the support team. Contact Support

To help us improve the quality of this article, please leave your email here so we can clarify further your feedback, if neccessary:
We will not send you spam or share your email address.

*This form is automated system. General questions, technical, sales, and product-related issues submitted through this form will not be answered.