To resolve this issue, the following steps may be done:
- Check the SG-SPS version under Trend Vision One console:
Click the image to enlarge.
- If your SPS version is already 2.0.1, then it should work well.
- If your SPS version is 2.0.0 then please try to upgrade it on the Cross Detection and Response (XDR) console.
- Usually this upgrade will finish within 30 minutes. If the upgrade does not finish, please login through the SG backend by following this KB article: How to log in to Service Gateway with root permission in Trend Vision One
- Check the SG running status with command “kubectl get pods -A“;, if the command returns the pods, then wait for the upgrade to finish. This is OK. A manual fix of the issue is needed, if the output displays similar to the following:
"internal error, please report: running "microk8s.kubectl" failed: cannot create transient scope: DBus error "System.Error.ENOSPC": [No space left on device]"
- If a manual fix is needed, execute the following command to delete the files:
cd /run/systemd/propagatem; rm -rf systemd-coredump@10* &&rm -rf systemd-coredump@11* && rm -rf systemd-coredump@12* && rm -rf systemd-coredump@13* && rm -rf systemd-coredump@14* && rm -rf systemd-coredump@15* && rm -rf systemd-coredump@16* && rm -rf systemd-coredump@1*; rm -rf systemd-coredump@20* &&rm -rf systemd-coredump@21* && rm -rf systemd-coredump@22* && rm -rf systemd-coredump@23* && rm -rf systemd-coredump@24* && rm -rf systemd-coredump@25* && rm -rf systemd-coredump@26* && rm -rf systemd-coredump@2*; rm -rf systemd-coredump@30* &&rm -rf systemd-coredump@31* && rm -rf systemd-coredump@32* && rm -rf systemd-coredump@33* && rm -rf systemd-coredump@34* && rm -rf systemd-coredump@35* && rm -rf systemd-coredump@36* && rm -rf systemd-coredump@3*; rm -rf systemd-coredump@40* &&rm -rf systemd-coredump@41* && rm -rf systemd-coredump@42* && rm -rf systemd-coredump@43* && rm -rf systemd-coredump@44* && rm -rf systemd-coredump@45* && rm -rf systemd-coredump@46* && rm -rf systemd-coredump@4*; rm -rf systemd-coredump@50* &&rm -rf systemd-coredump@51* && rm -rf systemd-coredump@52* && rm -rf systemd-coredump@53* && rm -rf systemd-coredump@54* && rm -rf systemd-coredump@55* && rm -rf systemd-coredump@56* && rm -rf systemd-coredump@5*; rm -rf systemd-coredump@60* &&rm -rf systemd-coredump@61* && rm -rf systemd-coredump@62* && rm -rf systemd-coredump@63* && rm -rf systemd-coredump@64* && rm -rf systemd-coredump@65* && rm -rf systemd-coredump@66* && rm -rf systemd-coredump@6*; rm -rf systemd-coredump@70* &&rm -rf systemd-coredump@71* && rm -rf systemd-coredump@72* && rm -rf systemd-coredump@73* && rm -rf systemd-coredump@74* && rm -rf systemd-coredump@75* && rm -rf systemd-coredump@76* && rm -rf systemd-coredump@7*; rm -rf systemd-coredump@80* &&rm -rf systemd-coredump@81* && rm -rf systemd-coredump@82* && rm -rf systemd-coredump@83* && rm -rf systemd-coredump@84* && rm -rf systemd-coredump@85* && rm -rf systemd-coredump@86* && rm -rf systemd-coredump@8*; rm -rf systemd-coredump@90* &&rm -rf systemd-coredump@91* && rm -rf systemd-coredump@92* && rm -rf systemd-coredump@93* && rm -rf systemd-coredump@94* && rm -rf systemd-coredump@95* && rm -rf systemd-coredump@96* && rm -rf systemd-coredump@9*;
- After executing the command above, check the SG status again with the command “kubectl get pods -A“;. This should now be working. You can now return to the XDR console and upgrade SG-SPS.
- If after executing the commands in Step 4 and Step 5 the issue still persists, then execute the command “microk8s inspect“ and ““journactl -u snap.microk8s.daemon-k8s-dqlite.service” to check the service status. You may get the following output:
fatal msg="Failed to start server: start node: raft_start(): io: load closed segment 0000000014232389-0000000014232597: entries batch 394 starting at byte 8384000: entries count in preamble is zero\n"
- According to the output message above, you have to recover the microk8s service. To do this, use the following command:
microk8s stop; rm /var/snap/microk8s/current/var/kubernetes/backend/0000000014232389-0000000014232597; microk8s start;
These files were taken from Step 6. - Once this is done, restart microK8s and then check the status using the command “kubectl get pods -A“;. You should be able to upgrade SG-SPS from Trend Vision One console again.