Help! DAOS files have been removed - the impact of a misconfigured backup job
Thomas Hampel
22 February 2023Recently a customer approached me with a request for help. I'd like to briefly share the story here because it was an interesting case.
On a Friday, the Domino team noticed severe problems with loading attachments, users reported they are no longer able to open attachments.
It seems like no single DAOS object can be opened anymore by the server.
Domino servers are reporting: Error 0x80070780: The file cannot be acessed by the system.
Checking the DAOS repository on the Domino server's disk revealed those files are displaying with a file size of XX MByte but actually have a size of ZERO BYTES (!!!)
Potential cause? Maybe a broken hard disc or filesystem? People even assumed Domino itself would be responsible for destroying DAOS objects on disk.
To mitigate the issue, a full restore of all DAOS objects was initaited which took a couple of hours. Afterwards it seemed the situation was resolved.
However just one day later the same problem appeared. All DAOS objects again had a size of 0 byte again with millions of DAOS objects being affected.
Root cause:
It turned out the backup software ( Commvault ) was misconfigured - instead of taking a backup of DAOS objects it was configured for >archiving< them.
Archiving in this case means that files will be moved to the backup environment but a 0 byte place holder will remain.
One could claim the user interface of Commvault backup easily allows for clicking the wrong option as both of them are listed next to each other.
There is no visible difference between the configuration screens later on, so unfortunately it was a human error/mistake to click on the wrong option.
Solution:
Initiate a restore job of files that were archived to the commvault envioronment.
https://documentation.commvault.com/v11/essential/134649_restoring_archived_data.html
Lessons learned:
Dont blame the top level application for a failure just because it is most impacted.
Open a support ticket at HCL and work together as a team to investigate and resolve the issue.
Tagged with: Administration Domino