Recovery of lost email/index objects

If lazy_expunge plugin is used, it's usually easy to undelete accidentally deleted mails because of it, even if entire folders are deleted using doveadm. However, this doesn't help if lazy_expunge isn't used, or if an administrator accidentally deletes the wrong users using doveadm obox user delete or in some other way directly from the object storage (e.g. with obox-fs(1)).

There is no direct function to recover the deleted objects in Dovecot and there should not be a need for it. However, admins are humans and things can happen by accident. Here's a process overview of what can be done in that catastrophic event to recover some of the mail data.

1. Contact your object storage provider, e.g. Scality. Object Storages normally only mark objects as deleted and the actual purge operations are usually cron based actions which happen some days after the actual deletion is marked for the objects.
2. Disable the purging operations from object storage according to object storage provider instructions.

If the objects were deleted only from the object storage (and not from Cassandra), trying to access the mails will cause Dovecot to log errors about unexpectedly lost mails/indexes. Dovecot won't automatically remove these from Cassandra, so undeleting the objects from the object storage will finish the recovery.

Dovecot based recovery

From Dovecot point of view there should be snapshot made of Cassandra which includes the deleted object references. This snapshot should be recovered so that it can be accessed again (an alternative might be to read the sstables snapshots directly from disk). If this is not the case, there are two more possibilities for data recovery:

• If deliver_log_format setting includes %{storage_id}, that expands to the object IDs. If the logs are still available, the object IDs for LMTP deliveries can be found from them. This won't help with mails saved via IMAP though.
• If HAProxy logs are available and have been configured to log X-Dovecot-Username, you can grep for that user's DELETEs.
• With Scality it's possible to list all index objects in the object storage (you can define a bitmask filter so it only returns e.g. folder index bundle objects). By HEADing these objects you can find out which users they belong to. This allows finding the index bundles for the users whose data needs to be recovered. The index bundles can be downloaded and unpacked with doveadm metacache unpack. The unpacked indexes can be doveadm dumped to find the list of email object IDs.

In any of the above cases the indexes most likely don't have the latest emails listed, so some of the newest mails will be lost. The only way to find them would be to list all objects in the object storage and try to find any mails whose object IDs aren't listed in any of the indexes. Those could be then HEADed to find out whether they should be recovered.

Possibility A for recovery:

1. Need to undelete in the object storage all the mail objects and the index bundle objects. Easiest is to use CQL to SELECT all the object IDs and undo the deletion based on it.
2. On snapshot-Cassandra: doveadm backup -u user mdbox:/tmp/user/mdbox - This might delete index bundles in production. Probably won't, at least commonly, but if it does happen, the production will start logging errors about unexpectedly missing objects. Those objects can then be again undeleted from the object storage if it happens.
3. On production-Cassandra: doveadm sync -u user mdbox:/tmp/user/mdbox - This will re-upload all the missing mail objects.
4. Delete again the old undeleted mail objects, since they were recreated already.

Possibility B for recovery:

1. Need to undelete in the object storage all the mail objects and the index bundle objects. Easiest is to use CQL to SELECT all the object IDs and undo the deletion based on it.
2. Copy all the undeleted mail objects and index objects from snapshot-Cassandra to production-Cassandra. There's generally no need to check if rows already exist. diff-bundles are a bit more troublesome: for them the "h" (host) field could be changed to append e.g. "-backup".
3. Access all the folders to make sure that dsync-merge works OK.