Deep Security Virtual Appliance (DSVA) has "Out of Disk Space" Event

Product / Version includes:

Deep Security11.0

Last updated: 2021/10/26

Solution ID: KA-0012293

Category: Troubleshoot

Summary

Customers using Deep Security Virtual Appliance version 11.0.211 (GM Build) may encounter continuous growth in the DSVA disk spaces resulting to high CPU utilization due to the ds_agent process keeps on performing clean up.

To identify this issue. You can check the following:
1. From the DSVA diagnostic package, check the dsa-state-capture-output.txt and search the keyword “Disk space critically”. You may see a similar message from the example below:

Aug 12 00:10:02 localhost root: Alert: Disk space low / : 94%
Aug 12 00:10:02 localhost root: WARNING: Disk space critically low on / : 94%
Aug 12 00:10:02 localhost root: Forcing clean on / : 94%

2. From the manager console, you can check the System Events for the following event:

-----------------------------------
The Agent/Appliance reported one or more warnings or errors. Details are found in the Agent/Appliance events listed below.

Agent/Appliance Event(s):

Time: August 11, 2021 23:35:21
Level: Error
Event ID: 6012
Event: Insufficient Disk Space
Description: The Agent/Appliance is running low on disk space and may not be able to store more events. Agent/Appliance file /var/opt/ds_agent resides on a volume with only 1,003 free MB. A minimum of 1,024 free MB is required.
----------------------------------

It has been identified as a problem with DSVA version 11.0.211 where all the log files under /var/opt/ds_agent/diag/dsva/ were not rotating properly, resulting in the growth in disk usage.

This issue was fixed in Deep Security Agent/DSVA 11.0 update 13 (Build 11.0.0-871)

To resolve this issue, you have three options:

Option 1: Re-deploy DSVA and upgrade to new Deep Security Agent(DSA) build

Download the above DSA 11.0 update 13 Build 11.0.0-871 and import to DSM server.
From the Computers Tab, locate the Relay server and click "Security Policy" from the action tab, then force the Relay Agent to sync the new DSA build from the DSM server immediately.
Delete the existing DSVA version 11.0.211 from the NSX-V Service deployment tab
Re-deploy DSVA again and upgrade it to version 11.0 update 13 Build 11.0.0-871.

Option 2: Manual delete logs files and upgrade to new Deep Security Agent(DSA) build

SSH or open DSVA terminal console from vCenter
Sign in and change permission via sudo -s with root permission
Execute the following command to delete the csv and log files under /var/opt/ds_agent/diag/dsva/
```
# rm -rf /var/opt/ds_agent/diag/dsva/*.csv
```
```
# rm -rf /var/opt/ds_agent/diag/dsva/*.log
```
Check the free disk space using command below and ensure that there is 1 GB available before upgrading the DSA.
```
# df -h
```
From the manager console, upgrade the DSVA to version 11.0 update 13 Build 11.0.0-871

Option 3: Manually change the file permission for log rotation.

SSH or open DSVA terminal console from vCenter
Sign in and change permission via sudo -s with root permission
Change the working path to /opt/ds_agent/dvfilter/ folder
Check the file "diag-logrotate.conf" permission it should be 664 before change
```
# ls -l /opt/ds_agent/dvfilter/diag-logrotate.conf
```
Manual change the file "diag-logrotate.conf" permission to 644
```
# chmod 644 /opt/ds_agent/dvfilter/diag-logrotate.conf
```

Execute the command to make the change effect immediately

# /usr/sbin/logrotate /opt/ds_agent/dvfilter/diag-logrotate.conf 2>/dev/null

Manual remove csv and log files under the /var/opt/ds_agent/diag/dsva/ folder

# rm -rf /var/opt/ds_agent/diag/dsva/*.log

# rm -rf /var/opt/ds_agent/diag/dsva/*.csv

Wait for few or 10 mins the csv and log files will generate automatically

Once the csv and log files created then force the log rotation via command

# /usr/sbin/logrotate --force /opt/ds_agent/dvfilter/diag-logrotate.conf

Then gz file will be created immediately

Was this article helpful?

Featured

Automation Center

Education Portal

Online Help Center

Service Status

TrendConnect

Deep Security Virtual Appliance (DSVA) has "Out of Disk Space" Event

Deep Security Virtual Appliance (DSVA) has "Out of Disk Space" Event

Product / Version includes:

Summary