Sign In with your
Trend Micro Account
Need Help?
Need More Help?

Create a technical support case if you need further support.

Deep Security Virtual Appliance (DSVA) has "Out of Disk Space" Event

    • Updated:
    • 20 Oct 2021
    • Product/Version:
    • Deep Security 11.0
    • Deep Security 11.0
    • Deep Security 11.0
    • Deep Security 11.0
    • Platform:
Summary
 Customers using Deep Security Virtual Appliance version 11.0.211 (GM Build) may encounter continuous growth in the DSVA disk spaces resulting to high CPU utilization due to the ds_agent process keeps on performing clean up. 

To identify this issue. You can check the following: 
1.    From the DSVA diagnostic package, check the dsa-state-capture-output.txt and search the keyword “Disk space critically”.  You may see a similar message from the example below: 
Aug 12 00:10:02 localhost root: Alert: Disk space low / : 94%
Aug 12 00:10:02 localhost root: WARNING: Disk space critically low on / : 94%
Aug 12 00:10:02 localhost root: Forcing clean on / : 94%

2.    From the manager console, you can check the System Events for the following event: 

-----------------------------------
The Agent/Appliance reported one or more warnings or errors. Details are found in the Agent/Appliance events listed below.

Agent/Appliance Event(s):

Time: August 11, 2021 23:35:21
Level: Error
Event ID: 6012
Event: Insufficient Disk Space
Description: The Agent/Appliance is running low on disk space and may not be able to store more events. Agent/Appliance file /var/opt/ds_agent resides on a volume with only 1,003 free MB. A minimum of 1,024 free MB is required.
----------------------------------

​​​​​​​
Details
Public
It has been identified as a problem with DSVA version 11.0.211 where all the log files under /var/opt/ds_agent/diag/dsva/ were not rotating properly, resulting in the growth in disk usage. 

This issue was fixed in Deep Security Agent/DSVA 11.0 update 13 (Build 11.0.0-871)

To resolve this issue, you have two options:
 
Option 1: Re-deploy DSVA and upgrade to new Deep Security Agent(DSA) build 
  1. Download the above DSA 11.0 update 13 Build 11.0.0-871 and import to DSM server. 
  2. From the Computers Tab, locate the Relay server and click "Security Policy" from the action tab, then force the Relay Agent to sync the new DSA build from the DSM server immediately.
  3. Delete the existing DSVA version 11.0.211 from the NSX-V Service deployment tab
  4. Re-deploy DSVA again and upgrade it to version 11.0 update 13 Build 11.0.0-871.
Option 2: Manual delete logs files and upgrade to new Deep Security Agent(DSA) build 
  1. SSH or open DSVA terminal console from vCenter
  2. Sign in and change permission via sudo -s with root permission 
  3. Execute the following command to delete the csv and log files under /var/opt/ds_agent/diag/dsva/
    # rm -rf /var/opt/ds_agent/diag/dsva/*.csv
    # rm -rf /var/opt/ds_agent/diag/dsva/*.log
  4. Check the free disk space using command below and ensure that there is 1 GB available before upgrading the DSA.
    # df -h
  5. From the manager console, upgrade the DSVA to version 11.0 update 13 Build 11.0.0-871

Option 3: Manually change the file permission for log rotation.
  1. SSH or open DSVA terminal console from vCenter
  2. Sign in and change permission via sudo -s with root permission 
  3. Change the working path to /opt/ds_agent/dvfilter/ folder
  4. Check the file "diag-logrotate.conf" permission it should be 664 before change
    # ls -l /opt/ds_agent/dvfilter/diag-logrotate.conf
  5. Manual change the file "diag-logrotate.conf" permission to 644
    # chmod 644 /opt/ds_agent/dvfilter/diag-logrotate.conf
  6. Execute the command to make the change effect immediately
    # /usr/sbin/logrotate /opt/ds_agent/dvfilter/diag-logrotate.conf 2>/dev/null 
  7. Manual remove csv and log files under the /var/opt/ds_agent/diag/dsva/ folder
    # rm -rf /var/opt/ds_agent/diag/dsva/*.log
    # rm -rf /var/opt/ds_agent/diag/dsva/*.csv
  8. Wait for few or 10 mins the csv and log files will generate automatically 
  9. Once the csv and log files created then force the log rotation via command 
    # /usr/sbin/logrotate --force /opt/ds_agent/dvfilter/diag-logrotate.conf 
  10. Then gz file will be created immediately
Premium
Internal
Partner
Rating:
Category:
Troubleshoot
Solution Id:
000288733
Feedback
Did this article help you?

Thank you for your feedback!


*This form is automated system. General questions, technical, sales, and product-related issues submitted through this form will not be answered.

If you need additional help, you may try to contact the support team. Contact Support

To help us improve the quality of this article, please leave your email here so we can clarify further your feedback, if neccessary:
We will not send you spam or share your email address.

*This form is automated system. General questions, technical, sales, and product-related issues submitted through this form will not be answered.