Sign In with your
Trend Micro Account
Need Help?
Need More Help?

Create a technical support case if you need further support.

Unable to install Deep Security 9.0 SP1 Patch 2 or Deep Security 9.5 Filter Driver on ESXi 5.5

    • Updated:
    • 24 Nov 2016
    • Product/Version:
    • Deep Security 9.5
    • Platform:
    • VMware ESXi 5.5

When preparing ESXi 5.5 for Deep Security Virtual Appliance (DSVA) 9.0 Service Pack (SP) 1 Patch 2 or 9.5 GM deployment, you get the following error during Filter Driver installation:

The installation transaction failed.

The following message appears in the ESXi esxupdate.log. The example logs below is from DSVA 9.0 SP1 Patch 2:

esxupdate: esxupdate: ERROR: InstallationError: ('Trend_bootbank_dvfilter-dsa_9.0.0-2636', '[Errno 4] Socket Error: [Errno 1] _ssl.c:1332: error:0906D06C:PEM routines:PEM_read_bio:no start line')


VMware has confirmed that this issue could happen when downloading and installing the Deep Security Filter Driver, or other vendors' VIB files, on ESXi 5.5.

This issue does not happen on ESXi 5.1 and older versions, because it is caused by a newly-added logic on ESXi 5.5. This new logic processes multiple partner CRLs in /usr/share/certs/vmparter.crl, but it does not clear the “PEM_R_NO_START_LINE” error in the openSSL error queue. This causes inaccurate communication during VIB payload downloading.

VMware will include the fix in ESXi 5.5 Update 2, which will be released in Q3 2014.

While waiting for the fix from VMware, do the following workaround:

  1. On the DSM, do the following:
    • For Windows: Open the Windows command line and change the directory path to C:\Program Files\Trend Micro\Deep Security Manager\. Execute the following command:

      dsm_c -action changesetting -name "settings.configuration.filterDriverNoSigCheck" -value true

    • For Linux: Log in via SSH and run the following command:

      /opt/dsm/dsm_c -action changesetting -name "settings.configuration.filterDriverNoSigCheck" -value true

    The DSM service will stop and start again during the process.

  2. Go to the DSM console and prepare the ESX again.

    The Filter Driver installation should now be successful.

If the above workaround does not work:
The procedure below is applicable for DSVA 9.0 only.
  1. Download
  2. Use the WinSCP tool to upload the file to the /tmp directory of the ESXi host.
  3. Use Putty to log in to the ESXi host via SSH.
  4. Go to the /tmp directory and run the following command:


    The following MD5 value will appear:


  5. Run the command “unzip”.
  6. Run the following command:

    esxcli software vib install --maintenance-mode -v /tmp/vib20/dvfilter-dsa/Trend_bootbank_dvfilter- dsa_9.0.0-2636.vib

    The following message will appear after 20-30 seconds:

    Installation Result
    Message: The update completed successfully, but the system needs to be rebooted for the changes
    to be effective.
    Reboot Required: true
    VIBs Installed: Trend_bootbank_dvfilter-dsa_9.0.0-2636

    Do not reboot the ESXi host yet.

  7. Go to the vCenter and click the host machine.
  8. Click Configuration > Networking.
  9. Under Standard Switch (vmservice-vswitch), click Properties > Add.
  10. Select Virtual Machine then click Next.
  11. Type "vmservice-trend-pg" as the Network Label.
  12. Click Next > Finish.

    The vmservice-trend-pg port group will be created.

  13. Reboot the ESXi host.
  14. Use Putty to log in to the ESXi host again and run the following commands to verify the installation:

    ~ # esxcli software vib list | grep Trend
    dvfilter-dsa                   9.0.0-995                             Trend   VMwareAccepted    2014-03-24
    ~ # vmkload_mod -l | grep dvfilter
    dvfilter                 12   144
    vmkapi_v2_0_0_0_dvfilter_shim1    8
    vmkapi_v2_1_0_0_dvfilter_shim0    8
    dvfilter-switch-security 1    192
    dvfilter-generic-fastpath0    180
    dvfilter-dsa             0    448

  15. Go to the DSM console and click Synchronize Now in the vCenter.

    ESXi will appear "Prepared" and vShield Endpoint will appear "Installed".

  16. Proceed with the usual DSVA deployment steps.
Deploy; Install; Upgrade
Solution Id:
Did this article help you?

Thank you for your feedback!

*This form is automated system. General questions, technical, sales, and product-related issues submitted through this form will not be answered.

If you need additional help, you may try to contact the support team. Contact Support

To help us improve the quality of this article, please leave your email here so we can clarify further your feedback, if neccessary:
We will not send you spam or share your email address.

*This form is automated system. General questions, technical, sales, and product-related issues submitted through this form will not be answered.