Policy deployment gets stuck in "Pending: Managed server deploying" status

Summary

The policy deployment from Apex Central to the Apex One features (e.g. Application Control, Vulnerability Protection, Endpoint Sensor, etc.) gets stuck in "Pending: Managed server deploying" status.

Root Cause Analysis

Errors can be seen in the the following logs:

"failed to get private key" error in Apex One ofcdebug.log located at..\Trend Micro\Apex One\PCCSRV\Log\:

2020 05/05 11:50:12 [144c : 2438] (00) (D) [][ofcservice.exe]
    getPFXFromCertificateStore - find one certificat matches the subject name=[OfcOSFWebApp] - 
    [libosfsvcclientutility.cpp(289)]

2020 05/05 11:50:12 [144c : 2438] (00) (E) [][ofcservice.exe]
    extractKeyPairFromPFX - failed to get private key, pkey is null - 
    [libosfsvcclientutility.cpp(442)]

"No client certificate, Authenticate failed" and "http response code=401" errors in Apex One ofcdebug.log located at..\Trend Micro\Apex One\PCCSRV\Log\:

2020 05/05 11:50:12 [68f4 : 0089] (00) (E) [-iAC-][w3wp.exe][SendAsync]
    [Mutual Authenticate] No client certificate, Authenticate failed, 
    url:([Apex One FQDN]/officescan_iac/osf/ONQUERY) - 
    [SecMsgHandler.cs(26)]

2020 05/05 11:50:12 [144c : 2438] (00) (E) [][ofcservice.exe]
    BoostHTTPClient::receive - http response code=401 - [libosfsvcclienthttpclient.cpp(101)]

Below are factors that can cause the policy deployment status to get stuck at "Pending: Managed server deploying" status:

The "failed to get private key" error in Apex One ofcdebug.log happens because OSF certificate private key cannot be exported in the environment. This can occur when the OSF certificates were replaced with 3rd-party certificates, but the "Mark this key as exportable" option was not enabled when the 3rd-party certificates were imported.
The "No client certificate, Authenticate failed" and "http response code=401" errors in Apex One ofcdebug.log can happen because of a misconfiguration in the SSL Settings for the OSFWebapp Site. The mutual authentication requires SSL, but if the SSL setting is misconfigured for the OSFWebapp site, it will return the HTTP 401 error.

Details

To fix the issue, perform the following steps:

EXPAND ALL

Step 1. Confirm if the OSF certificate was replaced by 3rd-party certificate

If it is confirmed that the OSF certificate was replaced by a 3rd-party certificate, do the following:

Remove the the certificate "OfcOSFWebApp" from "Trust People" store and "OfcOSF".
Import the 3rd-party certificate again, and enable "Mark this key as exportable" option.
For details on importing 3rd-party certificates, refer to this KB Article.

If the OSF certificate was not replaced, the existing cerficate should be removed, and a new set of certificate needs to be generated. Follow the steps below:

Manually delete the certificates:
- Delete "OfcOSFWebAppRootCA" certificate from "Trust Root".
- Delete "OfcOSFWebApp" certificate from "Trust People" and "OfcOSF".
Rebuild the certificate with the following command:
OfcSvcConfig.exe -FuncId InstallOSFCertificate -server_pccsrv_dir_path "C:\Program Files (x86)\Trend Micro\Apex One\PCCSRV" -osf_cert_password trend -apppoolname OfficeScanOSFAppPool -output_file "C:\Windows\OFCMAS1.LOG"

This command will set the password to "trend". It is recommended to change the password value to your preferred password.

Step 2. Enable SSL for the OSFWebapp site

Open the IIS Manager.
Ensure the setting "Require SSL" is enabled on the following sites.
- OfficeScan\osfwebapp\
- OfficeScan\officescan_iac\osf
Select Require SSL.
Select Accept under Client certificates.
Restart IIS service and deploy the policy again to verify the issue.