recovered-lost-folder-* folders

INFO

• There is an automated fixup script in the obox package: recover-mailbox(1)

Dovecot stores the list of folder GUID ↔ folder name mappings in dovecot.list.index and dovecot.list.index.log files as part of the user's root index bundle. If these files get lost or corrupted for any reason, the folder names may become lost.

Dovecot writes the folder names also to the folder indexes (dovecot.index*) directly when the folder is opened. This means that when a folder is renamed, the new name is written to dovecot.list.index but not immediately to the folder's own index - that's delayed until the folder is opened for the next time. When opening the folder even though the name is immediately written to dovecot.index* files, the folder index bundle won't be uploaded to object storage until something else forces it to be flushed. In any case, eventually all folders should have their (usually up-to-date) names in the folder index bundles.

When Dovecot finds that dovecot.list.index has become lost/corrupted, it starts fixing up the situation. First it lists all the folder GUIDs that exist in object storage and adds any missing ones to the dovecot.list.index. Initially it names them recovered-lost-folder-<folder GUID>. Then it opens the lost folders, which causes Dovecot to look for the latest known folder name from the dovecot.index*. If it exists, the folder is renamed to the previous name. In some situations the folder may already exist - for example a new INBOX may have been autocreated before the fixup was fully finished. In this situation the folder is renamed to the recovered-lost-folder-<folder GUID>, e.g. INBOX-recovered-lost-folder-123456.

These can be fixed manually with doveadm:

doveadm move -u user@domain INBOX-recovered-lost-folder-123456 mailbox INBOX all
doveadm mailbox delete -e -u user@domain INBOX
doveadm mailbox rename -u user@domain INBOX-recovered-lost-folder-123456 INBOX

The fixing is done in this direction, because the new INBOX folder is likely almost empty while the INBOX-recovered-lost-folder-123456 might have a lot of mails. So it avoids moving potentially a huge number of mails to INBOX. Also if the fix is done early enough, it preserves the INBOX's IMAP UIDs so clients won't re-download mails. The -e parameter to doveadm mailbox delete guarantees that the INBOX won't be deleted if there are any mails in it. It's possible that a newly arriving mail to INBOX causes the mailbox delete or rename to fail. In that case the commands need to be run again.

If the folder name doesn't exist in dovecot.index* yet, there's nothing in Dovecot indexes that can be used to find out its correct name. However, Dovecot often logs about rescanning indexes and those log lines contain both the folder name and the folder GUID. By looking at the log files it's possible to find the missing names and fix them. For example:

Mailbox Drafts: Rescanned index in 1@2/mailboxes/a6a03ff3d6ec9066b46013f4affb109a: 0 new mails, 0 mails lost, 0 kept

If there are folders whose original names can't be found, they should be made visible to the user by subscribing to them, and let the user figure out the name:

doveadm mailbox subscribe -u user@domain foldername

The recovered-lost-folder- prefix is configurable with obox_lost_mailbox_prefix setting.

There have been at least 4 known issues that causes the dovecot.list.index* corruption in the first place:

• Various generic bugs (usually related to race conditions with other processes) in Dovecot's index file handling that causes it to think they are corrupted. These should have been fixed.
• Accessing corrupted metacache after a server crash. Dovecot now correctly keeps track of server crashes and won't use potentially corrupted metacache.
• If metacache filesystem runs out of disk space, it's possible that the index file error handling code doesn't work perfectly and thinks that the indexes are corrupted instead. These should have been fixed.
• Using quota_over_status but WITHOUT quota_over_status_lazy_check = yes has a race condition that can cause the dovecot.list.index* to be deleted while they're still being used by other processes, which results in corruption. It would be a good idea to enable quota_over_status_lazy_check = yes in any case. See quota_over_status_lazy_check. This is still unfixed (DOP-128).

It's possible that there are still bugs left, so whenever this still happens, please send us all logs (at least error/warning logs, preferably also all logs that show concurrent imap/pop3/lmtp accesses). We can then try to figure out based on the logs what could have caused it.

Cassandra: Recovering folders invisible to force-resync

doveadm force resync (and folder list rebuilding in general) works by listing all rows for the user in the user_mailbox_index_diff_objects table. Normally this should list all folders that may have any mails. If for whatever reason a folder exists, but doesn't have a row in this table, it's invisible to the list rebuild. If a folder with a known GUID needs to be recovered, it can be done by inserting a row to the user_mailbox_index_diff_objects table:

INSERT INTO user_mailbox_index_diff_objects (u, g, h, m) VALUES ('USERID', 0xc92f64f79f0d1ed01e6d5b314f04886c, 'recoveryhost', '');

Afterwards doveadm force resync will find the folder and create an index for it. The "recoveryhost" string can be anything really, and its row can be deleted after doveadm force resync.

The main problem is how to find the GUIDs:

• Log files may contain the GUIDs when logging about rescanning a folder.
• If full text search indexes are enabled, downloading all the fts.D_* files and dumping them may show the missing GUIDs.
• It may be possible to find the user's old metacache files in one of the backends (where the user isn't currently active). Using doveadm dump dovecot.list.index will show the GUIDs.
• It may be possible to find the user's old root index bundle objects in some way, doveadm metacache unpack it and doveadm dump dovecot.list.index
  • For example: grep for PUTs for the user in HAProxy logs, filter out object IDs that aren't user root index bundles (see Scality (sproxyd) Object ID Format), undelete the object ID from Scality.
  • In a similar way, could also grep for folder index bundle PUTs, unpack the and doveadm dump to find its GUID from the index header
  • The above two cases are more complicated when the PUT is found for a diff bundle instead of base/self bundle. It might not be possible to unpack a diff bundle without its base being first unpacked.
• There's no CQL query that can return the missing folder GUIDs, but it would be possible to dump the Cassandra sstables and parse the (u, g) pairs. There are 3 possibilities:
  1. user_mailbox_index_objects : This table is relatively small, so it can be checked first. However, there's no guarantees that it will contain the missing GUIDs.
  2. user_mailbox_buckets : Again, this table is relatively small, but it's not filled out until a folder has more than 10k mails.
  3. user_mailbox_objects : This table definitely has the missing GUIDs, but it's huge and dumping and parsing it entirely may take a long time.