Views:
The VSS writer is automatically installed by SQL Express. It should not affect SMEX. To learn more about this application, refer to this Microsoft article on SQL Writer Service.

You may define a port value. Follow these steps:

  1. In SQL Server Configuration Manager, expand SQL Server Network Configuration in the console pane, and double-click Protocols for ScanMail.
  2. Double-click TCP/IP to show its properties.
  3. The TCP Dynamic Ports dialog box will show a value of 0, indicating the Database Engine is listening on dynamic ports. If you want to define a specific port, delete 0, and input your preferred TCP port. Click OK.

    config manager

  4. Modify dbcfg_DatabaseInstance.txt

    dbcfg_DatabaseInstance

  5. Open the service panel, right-click SQL Server (SCANMAIL) and then click Restart to stop and restart SQL Server. Restart SMEX Sevices.
 
For further information you may also refer to this Microsoft article, Configure a Server to Listen on a Specific TCP Port.

Instance SCANMAIL: Missing registry entry system\currentcontrolset\services\msolap$SCANMAIL
Instance SCANMAIL: Missing registry entry system\currentcontrolset\services\reportserver$SCANMAIL

SMEX doesn't use these hidden keys, but MSSQL$SCANMAIL should exist.

It is not ok to assign a different account for MSSQL$SCANMAIL.
The Shared memory of SQL instance is different from Shared memory folder of SMEX. When installing SQL instance, shared memory is enabled by default. It is better not to disable it because it can help for troubleshooting.
For more information, refer to this Microsoft article on SQL Server Network Configuration: Shared Memory Properties.
Upon testing it also shows that when disabling Shared memory of SQL instance without setting TCP port, the SQL instance service cannot be started.
Yes, it is ok. Please refer to the next item for detailed explanation.

There are three ways to find the certificate:

  • Exchange Control Panel(ECP). On the ECP website, navigate to servers and go to certificate.
  • Open a MMC > Add Certificates Snap In > Select Local Computer > Personal Folder > Certificates
  • Follow instructions on KB 1099200: Enabling SSL in ScanMail for Exchange. Open IIS Manager > Open Server Certificates.

Please refer to the Microsotf article on Enabling Encrypted Connections to the Database Engine.

To configure the server to force encrypted connections:

  1. In SQL Server Configuration Manager, expand SQL Server Network Configuration, right-click Protocols for server instance, and then select Properties.
  2. In the Protocols for instance name Properties dialog box, on the Certificate tab, select the desired certificate from the drop-down for the Certificate box, and then click OK.
  3. On the Flags tab, in the ForceEncryption box, select Yes, and then click OK to close the dialog box.
  4. Restart the SQL Server service and the SMEX master service.
 
  • The certificate must be issued for Server Authentication. The name of the certificate must be the Fully Qualified Domain Name (FQDN) of the computer, otherwise the certificate is not shown in the drop-down for the Certificate box of instance properties.
  • Enabling SSL encryption increases the security of data transmitted across networks between instances of SQL Server and SMEX. However, enabling encryption does slow performance. For reference refer to the Microsoft article on Encrypting Connections to SQL Server

The remote access enabled is default set when installing SQL Express. If there is no need to remote control SQL server db, it is ok to disable it.
Based on tests, there is a possibility that disabling it may affect visiting SQL Server Management Studio. Please contact Microsoft Technology Support for further information about this setting.

It is ok to disable it. SMEX doesn't add this user. It is added default when installing sql instance. Refer to the following Microsoft articles:

During setup of SQL Server Express a login is added for the BUILTIN\Users group. This allows all authenticated users of the computer to access the instance of SQL Server Express as a member of the public role. The BUILTIN\Users login can be safely removed to restrict Database Engine access to computer users who have individual logins or are members of other Windows groups with logins.
Before remove BUILTIN\Users, it is better to back up the BUILTIN\Users create sql script.