doveadm mail fs

The object storage can be used to access with doveadm-fs(1) and doveadm-mail-fs(1) commands. The doveadm-mail-fs(1) commands perform userdb lookup to get any user-specific (storage) settings.

How to get storage_user

storage_user is the user key which represents/identifies a user in cassandra tables for obox. This is typically some kind of a unique identifier, which is different from user@domain email address.

It is set in the userdb by replacing/overriding the user field. Below snippet is an example of replacing user field with a user attribute userUUID in LDAP configuration:

userdb ldap {
  ldap_filter = (&(mail=%{user})(objectClass=mailAccount))

  fields {
    # => user is replaced by the userUUID attribute in LDAP
    user = %{ldap:userUUID},
    quota_storage_size = %{ldap:quota}B
    ...
  }
}

If there is no such user replacement in the userdb, storage_user is identical to the user login id (typically mail address).

You can see the storage_user using doveadm user:

doveadm user -f user testuser@example.net
4e353905-a9fa-5aaa-974a-01d8f03aebf2

Common doveadm mail fs samples

Command line samples for mailbox

Iterating mailboxes (get list of mailbox guids)

You use doveadm mail fs iter-dirs with metacache parameter:

doveadm mail fs iter-dirs -u <username> metacache <storage_user>/mailboxes/

Command-line ex.1: if <username> is testuser@example.net and its storage_user is same as the username:

doveadm mail fs iter-dirs -u testuser@example.net metacache testuser@example.net/mailboxes/

Command-line ex.2: if <username> is testuser@example.net and its storage_user is 4e353905-a9fa-5aaa-974a-01d8f03aebf2:

doveadm mail fs iter-dirs -u testuser@example.net metacache 4e353905-a9fa-5aaa-974a-01d8f03aebf2/mailboxes/

Below is an example command line and its output when username is testuser@example.net and its storage_user has the UUID 4e353905-a9fa-5aaa-974a-01d8f03aebf2:

# For <object-path>, use "{storage_user}/mailboxes/"

doveadm mail fs iter-dirs -u testuser@example.net metacache 4e353905-a9fa-5aaa-974a-01d8f03aebf2/mailboxes/
08e3d21425d5a861b97c0000fb67425d
78519d25d9a0a861432d0000fb67425d
78a3011a1ed5a861b27c0000fb67425d
8068212321d5a861b57c0000fb67425d
b39cd138716aa961800c0000fb67425d

You see above that it prints mailbox-GUIDs (08e3d21425d5a861b97c0000fb67425d, 78519d25d9a0a861432d0000fb67425d, ...) for those mailboxes the user has.

Finding/Iterating mailbox index bundle(s) for a mailbox (then download to local and unpack)

• You use doveadm mail fs iter with metacache to iterate mailbox index-bundle(s),
• then you use doveadm mail fs get to download index-bundle from the iter output,
• you can unpack the index-bundle using doveadm metacache unpack, and then
• run doveadm dump -t index ....

Below is an example when username is testuser@example.net and its storage_user is in UUID 4e353905-a9fa-5aaa-974a-01d8f03aebf2:

# For <object-path>, use "{storage_user}/mailboxes/{mailbox-GUID}/idx/"

doveadm mail fs iter -u testuser@example.net metacache 4e353905-a9fa-5aaa-974a-01d8f03aebf2/mailboxes/2819a631168aa8611c7a0000fb67425d/idx/
bundle.61a893ab.4c6f8.c338a912ab93a861717b0000fb67425d.dovebe%2dbnd%2dcb001%2edovebe%2eipt.

The resulting bundle.61a893ab.4c6f8.c338a912ab93a861717b0000fb67425d.dovebe%2dbnd%2dcb001%2edovebe%2eipt. index-bundle is a self-bundle as implied by the trailing dot. See the fname-parse(1) script for a convenient way to find out the type and creation timestamp of a bundle.

You can download that index-bundle using doveadm mail fs get:

# For <object-path>, use "{storage_user}/mailboxes/{mailbox-GUID}/idx/{bundle-name}"

doveadm mail fs get -u testuser@example.net metacache \
    4e353905-a9fa-5aaa-974a-01d8f03aebf2/mailboxes/2819a631168aa8611c7a0000fb67425d/idx/bundle.61a893ab.4c6f8.c338a912ab93a861717b0000fb67425d.dovebe%2dbnd%2dcb001%2edovebe%2eipt. \
    > /tmp/tempfile-for-self-bundle

Then you can unpack the downloaded index-bundle to a temp directory, and run doveadm dump:

# create a directory where the index-bundle will be unpacked to
mkdir /tmp/unpack-dir
# use 'doveadm metacache unpack' to unpack bundle
doveadm metacache unpack /tmp/tempfile-for-self-bundle /tmp/unpack-dir
# then dump it
doveadm dump -t index /tmp/unpack-dir

When the mailbox has both base index-bundle and diff-index-bundle, both will be shown like below (mailbox-GUID 78519d25d9a0a861432d0000fb67425d).

# For <object-path>, use "{storage_user}/mailboxes/{mailbox-GUID}/idx/"

doveadm mail fs iter -u testuser@example.net metacache 4e353905-a9fa-5aaa-974a-01d8f03aebf2/mailboxes/78519d25d9a0a861432d0000fb67425d/idx/
bundle.61a8d665.2cd53.380cf20a65d6a8618b7d0000fb67425d

bundle.61a8d665.2cd53.380cf20a65d6a8618b7d0000fb67425d-e881a13576d7a8617b7d0000fb67425d.dovebe%2dbnd%2dcb001%2edovebe%2eipt is a diff-bundle.

You download both base- and diff- bundles using doveadm mail fs get:

# <object-path> is {storage_user}/mailboxes/{mailbox-GUID}/idx/{bundle-name}

# download base index-bundle
doveadm mail fs get -u testuser@example.net metacache \
    4e353905-a9fa-5aaa-974a-01d8f03aebf2/mailboxes/78519d25d9a0a861432d0000fb67425d/idx/bundle.61a8d665.2cd53.380cf20a65d6a8618b7d0000fb67425d > /tmp/tmp-for-base

# download diff index-bundle
doveadm mail fs get -u testuser@example.net metacache \
    4e353905-a9fa-5aaa-974a-01d8f03aebf2/mailboxes/78519d25d9a0a861432d0000fb67425d/idx/bundle.61a8d665.2cd53.380cf20a65d6a8618b7d0000fb67425d-e881a13576d7a8617b7d0000fb67425d.dovebe%2dbnd%2dcb001%2edovebe%2eip \
    > /tmp/tmp-for-diff

Then unpack them, in the order base-bundle then diff-bundle and doveadm dump:

mkdir /tmp/unpack-dir

# unpack base bundle
doveadm metacache unpack /tmp/tmp-for-base /tmp/unpack-dir

# unpack diff bundle on top of base (you cannot unpack diff alone if base is not unpacked yet)
doveadm metacache unpack /tmp/tmp-for-diff /tmp/unpack-dir

# then dump it
doveadm dump -t index /tmp/unpack-dir

doveadm mail fs stat and doveadm mail fs metadata to bundle

doveadm mail fs stat for index-bundle (it looks its output is size=... only):

# <object-path> is {storage_user}/mailboxes/{mailbox-GUID}/index/{bundle-name}    # same to doveadm mail fs get

doveadm mail fs stat -u testuser@example.net metacache \
    4e353905-a9fa-5aaa-974a-01d8f03aebf2/mailboxes/2819a631168aa8611c7a0000fb67425d/idx/bundle.61a893ab.4c6f8.c338a912ab93a861717b0000fb67425d.dovebe%2dbnd%2dcb001%2edovebe%2eipt.
4e353905-a9fa-5aaa-974a-01d8f03aebf2/mailboxes/2819a631168aa8611c7a0000fb67425d/idx/bundle.61a893ab.4c6f8.c338a912ab93a861717b0000fb67425d.dovebe%2dbnd%2dcb001%2edovebe%2eipt. size=768

doveadm mail fs metadata for index-bundle, metadata (:X-Dovecot-fs...., username, size, mailbox-guid, ...) is in the output:

# <object-path> is {storage_user}/mailboxes/{mailbox-GUID}/index/{bundle-name}    # same to doveadm mail fs get

doveadm mail fs metadata -u testuser@example.net metacache \
    4e353905-a9fa-5aaa-974a-01d8f03aebf2/mailboxes/2819a631168aa8611c7a0000fb67425d/idx/bundle.61a893ab.4c6f8.c338a912ab93a861717b0000fb67425d.dovebe%2dbnd%2dcb001%2edovebe%2eipt.
:/X-Dovecot-fs-api-OrigPath=4e353905-a9fa-5aaa-974a-01d8f03aebf2/mailboxes/2819a631168aa8611c7a0000fb67425d/idx/bundle.61a893ab.4c6f8.c338a912ab93a861717b0000fb67425d.dovebe%2dbnd%2dcb001%2edovebe%2eipt.
username=4e353905-a9fa-5aaa-974a-01d8f03aebf2
size=1680
mailbox-guid=2819a631168aa8611c7a0000fb67425d
fname=bundle.61a893ab.4c6f8.c338a912ab93a861717b0000fb67425d.dovebe%2dbnd%2dcb001%2edovebe%2eipt.

Command line samples for user index

Iterating user indexes

You use doveadm mail fs iter with metacache parameter:

# <object-path> is {storage_user}/idx/

doveadm mail fs iter -u testuser@example.net metacache 4e353905-a9fa-5aaa-974a-01d8f03aebf2/idx/
bundle.61a96b9e.25a31.684f30099e6ba961180c0000fb67425d.dovebe%2dbnd%2dcb001%2edovebe%2eipt.

This returns the self-bundle for the user-index.

Command line samples for mail objects

Iterating mail objects

INFO

You cannot get the bucket-id easily. If the number of mails in a mailbox is small, you can assume that bucket-id is zero. But if there are many mails in a mailbox (more than approximately 10.000), or after many mails have been expunged so that bucket-id is not zero you can not know what bucket-id a mail is in.

You use doveadm mail fs iter with obox parameter:

# <object-path> is {storage_user}/mailboxes/{mailbox-GUID}/{bucket-id}/

doveadm mail fs iter -u testuser@example.net obox \
    4e353905-a9fa-5aaa-974a-01d8f03aebf2/mailboxes/3839a60b4685a861bc790000fb67425d/0/
28d6f0154685a861bc790000fb67425d    # OID of mail object
487ea534bb97a861567c0000fb67425d
70c75739bd97a8615a7c0000fb67425d
b83dfb36ba97a861517c0000fb67425d
e02a950abd97a861587c0000fb67425d

Downloading a mail object

You use doveadm mail fs get with obox parameter:

# <object-path> is {storage_user}/mailboxes/{mailbox-GUID}/{bucket-id}/{OID}

doveadm mail fs get -u testuser@example.net obox \
  4e353905-a9fa-5aaa-974a-01d8f03aebf2/mailboxes/3839a60b4685a861bc790000fb67425d/0/28d6f0154685a861bc790000fb67425d > /tmp/mail-file

Command line samples for fts indexes

Iterating fts objects

You use doveadm mail fs iter with fts/dovecot parameter:

# <object-path> is "" (empty)

doveadm mail fs iter -u testuser@example.net fts/dovecot ""
fts.D_62fb830256d2a861c5560000fb67425d.00000174-000004cc.0001
fts.D_64e7d91038d2a861c5560000fb67425d.000003b3-00001358.0001
fts.D_6d6c562fb9d1a86169550000fb67425d.00000980-0000505c.0001
fts.D_792dea2e41d6a8617a7d0000fb67425d.000000ab-00000200.0001
fts.L_62fb830256d2a861c5560000fb67425d.00000174-000600ae.0001
fts.L_64e7d91038d2a861c5560000fb67425d.000003b3-001a02f2.0001
fts.L_6d6c562fb9d1a86169550000fb67425d.00000980-006e0c76.0001
fts.L_792dea2e41d6a8617a7d0000fb67425d.000000ab-0001cbeb.0001
fts.W_62fb830256d2a861c5560000fb67425d.00000174-00001ddc.0001
fts.W_64e7d91038d2a861c5560000fb67425d.000003b3-00001ddc.0001
fts.W_6d6c562fb9d1a86169550000fb67425d.00000980-00001ddc.0001
fts.W_792dea2e41d6a8617a7d0000fb67425d.000000ab-00001de8.0001

Downloading an fts object

You use doveadm mail fs get with fts/dovecot parameter:

# <object-path> is simply a fts object name
doveadm mail fs get -u testuser@example.net fts/dovecot fts.D_62fb830256d2a861c5560000fb67425d.00000174-000004cc.0001 > /tmp/fts-D-file