obox-fs.sh

The dovecot-pro-obox package comes with a wrapper script around doveadm-fs(1) that allows a more convenient way to handle the most common tasks. See obox-fs(1) for an up-to-date description of the offered operations.

How to get storage_user before running obox-fs.sh

storage_user is the user key which represents/identifies a user in cassandra tables for obox.

It is set in the userdb by replacing/overriding 'user' to a unique id which identify a user uniquely in the system. Below snippet is an example how replacing 'user' to a user attribute 'userUUID' in ldap configuration for userdb.

(ldap configuration for userdb)

...
user_filter = (&(mail=%u)(objectClass=mailAccount))

user_attrs = =user=%{ldap:userUUID},\    # => user is replaced by the userUUID attribute in LDAP
             =base_user=%{ldap:mail},\
             =quota_rule=*:storage=%{ldap:quota},\
             =quota_rule2=Trash:storage=+100M,\
             ...

If there is no such user replacement in the userdb, storage_user is identical to the user login id (typically mail address).

You get what storage_user is assigned to a user using doveadm user:

doveadm user testuser@example.net
field    value
user     4e353905-a9fa-5aaa-974a-01d8f03aebf2 => user is replaced to '4e353905-a9fa-5aaa-974a-01d8f03aebf2', and this is storage_id.
uid      500
gid      500
home     /var/dovecot/vmail/ce/4e353905-a9fa-5aaa-974a-01d8f03aebf2
mail     obox:4e353905-a9fa-5aaa-974a-01d8f03aebf2:INDEX=~/:CONTROL=~/:VOLATILEDIR=/dev/shm/dovecot/volatile/24/4e353905-a9fa-5aaa-974a-01d8f03aebf2:NO-NOSELECT
... (rest of output from 'doveadm user'

You can also use -f user (doveadm user -f user <user>) to get user only.

doveadm user -f user testuser@example.net
4e353905-a9fa-5aaa-974a-01d8f03aebf2

Common obox-fs.sh samples

Command line samples for mailbox

Iterating mailboxes (get list of mailbox guids)

You use iter-dirs, with -set obox_index_fs parameter:

obox-fs.sh -set obox_index_fs iter-dirs username <storage_user>/<mailboxes>

Command-line ex.1: if <username> is testuser@example.net and its storage_user is same as the username:

obox-fs.sh -set obox_index_fs iter-dirs testuser@example.net testuser@example.net/mailboxes

Command-line ex.2: if <username> is testuser@example.net and its storage_user is testuser (local part of email address):

obox-fs.sh -set obox_index_fs iter-dirs testuser@example.net testuser/mailboxes

Below is an example command line and its output when username is testuser@example.net and its storage_user has the UUID 4e353905-a9fa-5aaa-974a-01d8f03aebf2:

# For <object-path>, use "{storage_user}/mailboxes"

obox-fs.sh -set obox_index_fs iter-dirs testuser@example.net 4e353905-a9fa-5aaa-974a-01d8f03aebf2/mailboxes
08e3d21425d5a861b97c0000fb67425d
78519d25d9a0a861432d0000fb67425d
78a3011a1ed5a861b27c0000fb67425d
8068212321d5a861b57c0000fb67425d
b39cd138716aa961800c0000fb67425d

You see above that it prints mailbox-GUIDs (08e3d21425d5a861b97c0000fb67425d, 78519d25d9a0a861432d0000fb67425d, ...) for those mailboxes the user has.

Finding/Iterating mailbox index bundle(s) for a mailbox (then download to local and unpack)

• You use iter with -set obox_index_fs to iterate mailbox index-bundle(s),
• then you use get to download index-bundle from the iter output,
• you can unpack the index-bundle using doveadm metacache unpack, and then
• run doveadm dump -t index ....

Below is an example when username is testuser@example.net and its storage_user is in UUID 4e353905-a9fa-5aaa-974a-01d8f03aebf2.

self-bundle is in its output:

# For <object-path>, use "{storage_user}/mailboxes/{mailbox-GUID}/idx"

obox-fs.sh -set obox_index_fs iter testuser@example.net 4e353905-a9fa-5aaa-974a-01d8f03aebf2/mailboxes/2819a631168aa8611c7a0000fb67425d/idx
bundle.61a893ab.4c6f8.c338a912ab93a861717b0000fb67425d.dovebe%2dbnd%2dcb001%2edovebe%2eipt.

The resulting bundle.61a893ab.4c6f8.c338a912ab93a861717b0000fb67425d.dovebe%2dbnd%2dcb001%2edovebe%2eipt. index-bundle is a self-bundle as implied by the trailing dot. See the fname-parse(1) script for a convenient way to find out the type and creation timestamp of a bundle.

You can download that index-bundle using obox-fs.sh get:

# For <object-path>, use "{storage_user}/mailboxes/{mailbox-GUID}/idx/{bundle-name}"

obox-fs.sh -set obox_index_fs get testuser@example.net \
    4e353905-a9fa-5aaa-974a-01d8f03aebf2/mailboxes/2819a631168aa8611c7a0000fb67425d/idx/bundle.61a893ab.4c6f8.c338a912ab93a861717b0000fb67425d.dovebe%2dbnd%2dcb001%2edovebe%2eipt. \
    > /tmp/tempfile-for-self-bundle

Then you can unpack the downloaded index-bundle to a temp directory, and run doveadm dump:

# create a directory where the index-bundle will be unpacked to
mkdir /tmp/unpack-dir
# use 'doveadm metacache unpack' to unpack bundle
doveadm metacache unpack /tmp/tempfile-for-self-bundle /tmp/unpack-dir
# then dump it
doveadm dump -t index /tmp/unpack-dir

When the mailbox has both base index-bundle and diff-index-bundle, both will be shown like below (mailbox-GUID 78519d25d9a0a861432d0000fb67425d).

# For <object-path>, use "{storage_user}/mailboxes/{mailbox-GUID}/idx/"

obox-fs.sh -set obox_index_fs iter testuser@example.net 4e353905-a9fa-5aaa-974a-01d8f03aebf2/mailboxes/78519d25d9a0a861432d0000fb67425d/idx/
bundle.61a8d665.2cd53.380cf20a65d6a8618b7d0000fb67425d

bundle.61a8d665.2cd53.380cf20a65d6a8618b7d0000fb67425d-e881a13576d7a8617b7d0000fb67425d.dovebe%2dbnd%2dcb001%2edovebe%2eipt is a diff-bundle.

You download both base- and diff- bundles using obox-fs.sh get:

# <object-path> is {storage_user}/mailboxes/{mailbox-GUID}/idx/{bundle-name}

# download base index-bundle
obox-fs.sh -set obox_index_fs iter testuser@example.net \
    4e353905-a9fa-5aaa-974a-01d8f03aebf2/mailboxes/78519d25d9a0a861432d0000fb67425d/idx/bundle.61a8d665.2cd53.380cf20a65d6a8618b7d0000fb67425d > /tmp/tmp-for-base

# download diff index-bundle
obox-fs.sh -set obox_index_fs iter testuser@example.net \
    4e353905-a9fa-5aaa-974a-01d8f03aebf2/mailboxes/78519d25d9a0a861432d0000fb67425d/idx/bundle.61a8d665.2cd53.380cf20a65d6a8618b7d0000fb67425d-e881a13576d7a8617b7d0000fb67425d.dovebe%2dbnd%2dcb001%2edovebe%2eip \
    > /tmp/tmp-for-diff

Then unpack them, in the order base-bundle then diff-bundle and doveadm dump:

mkdir /tmp/unpack-dir

# unpack base bundle
doveadm metacache unpack /tmp/tmp-for-base /tmp/unpack-dir

# unpack diff bundle on top of base (you cannot unpack diff alone if base is not unpacked yet)
doveadm metacache unpack /tmp/tmp-for-diff /tmp/unpack-dir

# then dump it
doveadm dump -t index /tmp/unpack-dir

obox-fs.sh stat and obox-fs.sh metadata to bundle

obox-fs.sh get for index-bundle (it looks its output is size=... only):

# <object-path> is {storage_user}/mailboxes/{mailbox-GUID}/index/{bundle-name}    # same to obox-fs.sh get

obox-fs.sh -set obox_index_fs stat testuser@example.net \
    4e353905-a9fa-5aaa-974a-01d8f03aebf2/mailboxes/2819a631168aa8611c7a0000fb67425d/idx/bundle.61a893ab.4c6f8.c338a912ab93a861717b0000fb67425d.dovebe%2dbnd%2dcb001%2edovebe%2eipt.
4e353905-a9fa-5aaa-974a-01d8f03aebf2/mailboxes/2819a631168aa8611c7a0000fb67425d/idx/bundle.61a893ab.4c6f8.c338a912ab93a861717b0000fb67425d.dovebe%2dbnd%2dcb001%2edovebe%2eipt. size=768

obox-fs.sh metadata for index-bundle, metadata (:X-Dovecot-fs...., username, size, mailbox-guid, ...) is in the output:

# <object-path> is {storage_user}/mailboxes/{mailbox-GUID}/index/{bundle-name}    # same to obox-fs.sh get

obox-fs.sh -set obox_index_fs metadata testuser@example.net \
    4e353905-a9fa-5aaa-974a-01d8f03aebf2/mailboxes/2819a631168aa8611c7a0000fb67425d/idx/bundle.61a893ab.4c6f8.c338a912ab93a861717b0000fb67425d.dovebe%2dbnd%2dcb001%2edovebe%2eipt.
:/X-Dovecot-fs-api-OrigPath=4e353905-a9fa-5aaa-974a-01d8f03aebf2/mailboxes/2819a631168aa8611c7a0000fb67425d/idx/bundle.61a893ab.4c6f8.c338a912ab93a861717b0000fb67425d.dovebe%2dbnd%2dcb001%2edovebe%2eipt.
username=4e353905-a9fa-5aaa-974a-01d8f03aebf2
size=1680
mailbox-guid=2819a631168aa8611c7a0000fb67425d
fname=bundle.61a893ab.4c6f8.c338a912ab93a861717b0000fb67425d.dovebe%2dbnd%2dcb001%2edovebe%2eipt.

Command line samples for user index

Iterating user indexes

You use iter, with -set obox_index_fs parameter:

# <object-path> is {storage_user}/idx

obox-fs.sh -set obox_index_fs iter testuser@example.net 4e353905-a9fa-5aaa-974a-01d8f03aebf2/idx
bundle.61a96b9e.25a31.684f30099e6ba961180c0000fb67425d.dovebe%2dbnd%2dcb001%2edovebe%2eipt.

This returns the self-bundle for the user-index.

Command line samples for mail objects

Iterating mail objects

You use iter, with -set obox_fs parameter:

# <object-path> is {storage_user}/mailboxes/{mailbox-GUID}

obox-fs.sh iter testuser@example.net 4e353905-a9fa-5aaa-974a-01d8f03aebf2/mailboxes/3839a60b4685a861bc790000fb67425d
28d6f0154685a861bc790000fb67425d    # OID of mail object
487ea534bb97a861567c0000fb67425d
70c75739bd97a8615a7c0000fb67425d
b83dfb36ba97a861517c0000fb67425d
e02a950abd97a861587c0000fb67425d

Downloading a mail object

You use get, with -set obox_fs parameter:

NOTE

You cannot get the bucket-id. If the number of mails in a mailbox is small you can assume that bucket-id is zero. But if there are many mails in a mailbox (more than approximately 10.000), or after many mails have been expunged so that bucket-id is not zero you can not know what bucket-id a mail is in.

# <object-path> is {storage_user}/mailboxes/{mailbox-GUID}/{bucket-id}/{OID}

obox-fs.sh get testuser@example.net 4e353905-a9fa-5aaa-974a-01d8f03aebf2/mailboxes/3839a60b4685a861bc790000fb67425d/0/28d6f0154685a861bc790000fb67425d > /tmp/mail-file

Command line samples for fts indexes

Iterating fts objects

You use iter, with -set fts_dovecot_fs parameter:

# <object-path> is "" (empty)

obox-fs.sh -set fts_dovecot_fs iter testuser@example.net ""
fts.D_62fb830256d2a861c5560000fb67425d.00000174-000004cc.0001
fts.D_64e7d91038d2a861c5560000fb67425d.000003b3-00001358.0001
fts.D_6d6c562fb9d1a86169550000fb67425d.00000980-0000505c.0001
fts.D_792dea2e41d6a8617a7d0000fb67425d.000000ab-00000200.0001
fts.L_62fb830256d2a861c5560000fb67425d.00000174-000600ae.0001
fts.L_64e7d91038d2a861c5560000fb67425d.000003b3-001a02f2.0001
fts.L_6d6c562fb9d1a86169550000fb67425d.00000980-006e0c76.0001
fts.L_792dea2e41d6a8617a7d0000fb67425d.000000ab-0001cbeb.0001
fts.W_62fb830256d2a861c5560000fb67425d.00000174-00001ddc.0001
fts.W_64e7d91038d2a861c5560000fb67425d.000003b3-00001ddc.0001
fts.W_6d6c562fb9d1a86169550000fb67425d.00000980-00001ddc.0001
fts.W_792dea2e41d6a8617a7d0000fb67425d.000000ab-00001de8.0001

Downloading an fts object

You use get, with -set fts_dovecot_fs parameter:

# <object-path> is simply a fts object name
obox-fs.sh -set fts_dovecot_fs get testuser@example.net fts.D_62fb830256d2a861c5560000fb67425d.00000174-000004cc.0001 > /tmp/fts-D-file