Deduplicate email objects via storage

In situations where there are tens of millions of emails in a single folder obox might crash. To start fixing up the situation safely, duplicate GUIDs can be deleted directly using the object storage operations:

WARNING

It may not be safe to run 5-8) multiple times without some further changes, because on the next run it might decide to leave a different already deleted mail object as the non-deleted one, causing all the mails to be deleted. Also it would try to re-delete already deleted mails. This could be fixable by updating all-mails.txt and deleting metadata.* for mails that were deleted.

TIP

To make sure this procedure isn't a waste of time (and to monitor progress), you can run steps 5-6) while 4) is still running. You can see in duplicate-guids.txt where it's finding lots of duplicates or not. Once step 4) is finished, run 5-6) steps again and finish with 7-8).

This procedure makes use of the obox-fs(1) script that is available in the dovecot-pro-obox package:

1. cd into some directory which you won't lose easily. This will contain a local copy of all the mails' metadata.
2. Get list of all mail objects:

   obox-fs.sh iter user@example user@example/mailboxes/$mailbox_guid > all-mails.txt

3. Verify that all-mails.txt contains all of the relevant entries (compare lines in file with number of emails):

   wc -l all-mails.txt

4. Save metadata for all mail objects locally (this could take days):

   while read oid; do
       obox-fs.sh metadata user@example user@example/mailboxes/$mailbox_guid/$oid > metadata.$oid
   done < all-mails.txt

   While it's running, verify that the metadata.* files' contents look reasonable. Especially verify that they contain a line such as: "guid=00410a00f49fa463fe5e2c0071396a78"
5. Get a list of "email-GUID OID" lines sorted by GUIDs:

   find . -name 'metadata.*' |
   while read fname; do
       grep '^guid=' $fname | sed 's/guid=//' | tr -d '\n'
       echo $fname | sed 's:^\./metadata\.: :'
   done |
   sort > guid-oid.txt

6. Show a list of GUIDs that exist multiple times as OIDs:

   cat guid-oid.txt | cut -d ' ' -f 1 | sort | uniq -cd > duplicate-guids.txt

   You can also see from the first number how many such duplicates there are.
7. Get a list of duplicate OIDs that need to be deleted. One of the OIDs is left undeleted:

   cat duplicate-guids.txt | awk '{print $2}' |
   while read guid; do
       grep "^$guid " guid-oid.txt | sed "s/^$guid //" | tail -n +2
   done > delete-oids.txt

8. Delete these duplicate OIDs:

   cat delete-oids.txt |
   while read oid; do
       obox-fs.sh delete user@example user@example/mailboxes/$mailbox_guid/$oid
   done

   You may also want to use obox-fs.sh metadata and obox-fs.sh get commands to save a backup of the mail before deleting it, in case anything went wrong.