Search K
Appearance
Obox Storage Support: Scality (sproxyd)
Configuration
Scality sproxyd uses the sproxyd scheme for configuration:
plugin {
# Basic configuration:
obox_fs = sproxyd:http://scality.example.com/?parameters
}URL Formatting
An HTTP URL to specify how the object storage is accessed.
The parameters are specified as URL-style parameters, such as http://url/?param1=value1¶m2=value2.
URL escaping is used, so if password is foo/bar the URL is http://user:foo%2fbar@example.com/.
Additionally, because Dovecot expands %variables inside the plugin section, the % needs to be escaped. So the final string would be e.g.:
plugin {
obox_fs = s3:https://user:foo%%2fbar@example.com/ # password is foo/bar
}Configuration Parameters
| Parameter | Description | Default |
|---|---|---|
absolute_timeout=<time_msecs> | Maximum total time for an HTTP request to finish. Overrides all timeout configuration. | None |
addhdr=<name>:<value> | Add the specified header to all HTTP requests. | None |
addhdrvar=<name>:<variable> | Add the specified header to all HTTP requests and set the value to the expanded variables value. | None |
avoid_423=<time_msecs> | Using this setting allows to delay DELETE requests if the same object ID has been GET/HEAD/PUT by the same process within time_msecs. This is intended to reduce "423 Locked" sent by Scality. When this setting is disabled, no delay is added. Normally this setting should not be used. It should be only be set if it can be seen to bring a benefit. Careful investigation of current error-rates and consideration of the overall throughput of the platform are recommended before using it. | None |
bulk_delete_limit=<n> | Number of deletes supported within the same bulk delete request. 0 disables bulk deletes. | 1000 |
by-path=1 | Objects are accessed by path instead of by object ID. Scality sproxyd internally converts the paths into object IDs. This shouldn't normally be used. | 0 |
class=<n> | Scality Class of Service. 2 means that the objects are written to the Scality RING 3 times in total. This is generally the minimum allowable redundancy for mail and index objects. FTS data is more easily reproducible, so losing those indexes is not as critical; Class of Service 1 may be appropriate based on customer requirements. | None |
connect_timeout=<time_msecs> | Timeout for establishing a TCP connection. | <timeout> parameter |
delete_max_retries=<n> | Max number of HTTP request retries for delete actions. | <max_retries> parameter |
delete_timeout=<time_msecs> | Timeout for sending a delete HTTP response. | <timeout> parameter |
loghdr=<name> | Headers with the given name in HTTP responses are logged as part of any error, debug or warning messages related to the HTTP request. These headers are also included in the http_request_finished event as fields prefixed with http_hdr_. Can be specified multiple times. | None |
max_connect_retries=<n> | Number of connect retries. | 2 |
max_retries=<n> | Max number of HTTP request retries. Retries happen for 5xx errors as well as for 423(locked) with sproxyd. There is a wait between attempting next retry. The initial retry is done after 50ms. The following retries are done after waiting ten times as long as the previous attempt, so 50ms -> 500 ms -> 5s ->10s. The maximum wait time per attempt before retry is limited to 10 seconds. Please note that if the overall request time exceeds the configured absolute_timeout it takes precedence, emits an error and prevents further retries. While the configured timeout value determines how long HTTP responses are allowed to take before an error is emitted. | 4 |
no_trace_headers=1 | Set to 1 to not add X-Dovecot-User or X-Dovecot-Session headers to HTTP request. These headers are useful to correlate object storage requests to App Suite/Dovecot sessions. If not doing correlations via log aggregation, this is safe to disable. | 0 |
read_max_retries=<n> | Max number of HTTP request retries for read actions. | <max_retries> parameter |
read_timeout=<time_msecs> | Timeout for a receiving read HTTP response. | <timeout> parameter |
reason_header_max_length=<n> | Maximum length for X-Dovecot-Reason HTTP header. If header is present, it contains information why obox operation is being done. | 0 |
slow_warn=<time_msecs> | Log a warning about any HTTP request that takes longer than this time. | 5s |
timeout=<time_msecs> | Default timeout for HTTP responses, unless overwritten by other parameters. | 10s |
write_max_retries=<n> | Max number of HTTP request retries for write actions. | <max_retries> parameter |
write_timeout=<time_msecs> | Timeout for a write HTTP response. | <timeout> parameter |
Example Configuration
WARNING
All text indicated by {{VARIABLE NAME}} in the examples below MUST be replaced with your local configuration value(s).
Note
Dictmap must also be configured to use this storage driver.
mail_location = obox:%u:INDEX=~/:CONTROL=~/
plugin {
# SPROXYD_SCALITY_URL: The URL to the sproxyd interface.
# Examples use Class of Service (CoS) 2 for mail, CoS 1 for FTS;
# data compressed with zstd (level 3)
# Without lazy_expunge plugin:
obox_fs = fscache:512M:/var/cache/mails/%4Nu:compress:zstd:3:dictmap:proxy:dict-async:cassandra ; sproxyd:http://{{SPROXYD_SCALITY_URL}}/?class=2&reason_header_max_length=200 ; refcounting-table:lockdir=/tmp:bucket-size=10000:bucket-cache=%h/buckets.cache:nlinks-limit=3:delete-timestamp=+10s:bucket-deleted-days=11
# With lazy_expunge plugin:
#obox_fs = fscache:512M:/var/cache/mails/%4Nu:compress:zstd:3:dictmap:proxy:dict-async:cassandra ; sproxyd:http://{{SPROXYD_SCALITY_URL}}/?class=2&reason_header_max_length=200 ; refcounting-table:bucket-size=10000:bucket-cache=%h/buckets.cache:nlinks-limit=3:delete-timestamp=+10s:bucket-deleted-days=11
obox_index_fs = compress:zstd:3:dictmap:proxy:dict-async:cassandra ; sproxyd:http://{{SPROXYD_SCALITY_URL}}/?class=2&reason_header_max_length=200 ; diff-table
# It may be OK to store fewer copies of FTS indexes(class=1), because they can
# be rebuilt if they become completely lost.
fts_dovecot_fs = fts-cache:fscache:512M:/var/cache/fts/%4Nu:compress:zstd:3:dictmap:proxy:dict-async:cassandra ; sproxyd:http://{{SPROXYD_SCALITY_URL}}/?class=1&reason_header_max_length=200 ; dict-prefix=%u/fts/
}sproxyd API
See Scality sproxyd.
sproxyd Object ID Format
Dovecot uses its own Scality key format, which encodes the object type also to the key itself.
Key Format (160 bits)
| Size | Description |
|---|---|
| 152 bits | Entropy |
| 4 bits | Class |
| 4 bits | Replica |
"Entropy" Component
Definition of entropy component of Scality's Universal Key Scheme (UKS):
| Size | Description |
|---|---|
| 24 bits | Dispersion (important) |
| 64 bits | Object ID |
| 32 bits | Volume ID |
| 8 bits | Service ID (important) |
| 24 bits | App-specific |
So if we use 128 bit MD5 of the GUID as our unique key, this means we can do:
| Size | Description |
|---|---|
| 8 bits | Service ID = FS_SPROXYD_SERVICE_ID_DOVECOT = 0x83 |
| 8 bits | MD5 part suffix |
| 8 bits | Object type (see below) |
| 8 bits | Unused (0x00) |
| 4 bits | Class ID (configurable) |
| 4 bits | Replica ID (0x00) |
Where "object type" is:
| Internal Type | Value |
|---|---|
FS_SPROXYD_OBJECT_TYPE_UNKNOWN | 0x00 |
FS_SPROXYD_OBJECT_TYPE_MAIL | 0x01 |
FS_SPROXYD_OBJECT_TYPE_OLD_FTS_INDEX | 0x02 |
FS_SPROXYD_OBJECT_TYPE_FTS_INDEX | 0x03 |
FS_SPROXYD_OBJECT_TYPE_USER_INDEX_SELF | 0x08 | 0x01 |
FS_SPROXYD_OBJECT_TYPE_USER_INDEX_BASE | 0x08 | 0x02 |
FS_SPROXYD_OBJECT_TYPE_USER_INDEX_DIFF | 0x08 | 0x03 |
FS_SPROXYD_OBJECT_TYPE_BOX_INDEX_SELF | 0x08 | 0x04 | 0x01 |
FS_SPROXYD_OBJECT_TYPE_BOX_INDEX_BASE | 0x08 | 0x04 | 0x02 |
FS_SPROXYD_OBJECT_TYPE_BOX_INDEX_DIFF | 0x08 | 0x04 | 0x03 |
For listing index objects, these can be helpful masks:
| Internal Type | Value |
|---|---|
FS_SPROXYD_OBJECT_TYPE_BIT_INDEX | 0x08 |
FS_SPROXYD_OBJECT_TYPE_BIT_BOX_INDEX | 0x04 |
FS_SPROXYD_OBJECT_TYPE_BIT_DIFF_OR_SELF | 0x01 |
Utility Script
The scality-keys(1) script installed with the obox package takes a 160bit hex-encoded sproxyd Scality ID as input and outputs the object type.
Debug Headers
Dovecot sends the following HTTP headers towards storage. They should be logged for troubleshooting purposes:
X-Dovecot-UsernameX-Dovecot-Session-IdX-Dovecot-Reason
Storage-Side Object Metadata
When saving data to object storage, Dovecot stores metadata associated with each blob for data recovery purposes.
This data is written to the HTTP endpoint by adding Dovecot metadata headers to the request. When retrieving a message from object storage, this data is returned in the received headers (only parsed by Dovecot if needed).
For S3, the header names are: X-Object-meta-dovecot-<key>.
Email Objects
| Key | Description | Max Length (in bytes) | Other Info |
|---|---|---|---|
fname | Dovecot filename | N/A (installation dependent; username component to naming) | |
guid | Message GUID | 32 | |
origbox | Folder GUID of first folder where stored | 32 | Copying does not update |
pop3order | POP3 message order | 10 | Only if needed by migration |
pop3uidl | POP3 UIDL | N/A (depends on source installation) | Only if message was migrated |
received | Received data | 20 (in theory; rarely more than 10) | UNIX timestamp format |
saved | Saved data | 20 (in theory; rarely more than 10) | UNIX timestamp format |
size | Message size | 20 (in theory; rarely more than 10) | Size in bytes |
username | Dovecot unique username | N/A (installation dependent) |
Index Objects
| Key | Description | Max Length (in bytes) | Other Info |
|---|---|---|---|
fname | Dovecot filename | N/A (installation dependent; username component to naming) | |
mailbox-guid | Mailbox GUID the index refers to | 32 | |
size | Message size | 20 (in theory; rarely more than 10) | Size in bytes |
username | Dovecot unique username | N/A (installation dependent) |
FTS Index Objects
| Key | Description | Max Length (in bytes) | Other Info |
|---|---|---|---|
fname | Dovecot filename | N/A (installation dependent; username component to naming) | |
username | Dovecot unique username | N/A (installation dependent) |