Obox Storage Support: Scality (sproxyd)

Configuration

Scality sproxyd uses the sproxyd scheme for configuration:

plugin {
  # Basic configuration:
  obox_fs = sproxyd:http://scality.example.com/?parameters
}

URL Formatting

An HTTP URL to specify how the object storage is accessed.

The parameters are specified as URL-style parameters, such as http://url/?param1=value1&param2=value2.

URL escaping is used, so if password is foo/bar the URL is http://user:foo%2fbar@example.com/.

Additionally, because Dovecot expands %variables inside the plugin section, the % needs to be escaped. So the final string would be e.g.:

plugin {
  obox_fs = s3:https://user:foo%%2fbar@example.com/ # password is foo/bar
}

Configuration Parameters

Parameter	Description	Default
absolute_timeout=<time_msecs>	Maximum total time for an HTTP request to finish. Overrides all timeout configuration.	None
addhdr=<name>:<value>	Add the specified header to all HTTP requests.	None
addhdrvar=<name>:<variable>	Add the specified header to all HTTP requests and set the value to the expanded variables value.	None
avoid_423=<time_msecs>	Using this setting allows to delay DELETE requests if the same object ID has been GET/HEAD/PUT by the same process within time_msecs. This is intended to reduce "423 Locked" sent by Scality. When this setting is disabled, no delay is added. Normally this setting should not be used. It should be only be set if it can be seen to bring a benefit. Careful investigation of current error-rates and consideration of the overall throughput of the platform are recommended before using it.	None
bulk_delete_limit=<n>	Number of deletes supported within the same bulk delete request. 0 disables bulk deletes.	1000
by-path=1	Objects are accessed by path instead of by object ID. Scality sproxyd internally converts the paths into object IDs. This shouldn't normally be used.	0
class=<n>	Scality Class of Service. 2 means that the objects are written to the Scality RING 3 times in total. This is generally the minimum allowable redundancy for mail and index objects. FTS data is more easily reproducible, so losing those indexes is not as critical; Class of Service 1 may be appropriate based on customer requirements.	None
connect_timeout=<time_msecs>	Timeout for establishing a TCP connection.	<timeout> parameter
delete_max_retries=<n>	Max number of HTTP request retries for delete actions.	<max_retries> parameter
delete_timeout=<time_msecs>	Timeout for sending a delete HTTP response.	<timeout> parameter
loghdr=<name>	Headers with the given name in HTTP responses are logged as part of any error, debug or warning messages related to the HTTP request. These headers are also included in the http_request_finished event as fields prefixed with http_hdr_. Can be specified multiple times.	None
max_connect_retries=<n>	Number of connect retries.	2
max_retries=<n>	Max number of HTTP request retries. Retries happen for 5xx errors as well as for 423(locked) with sproxyd. There is a wait between attempting next retry. The initial retry is done after 50ms. The following retries are done after waiting ten times as long as the previous attempt, so 50ms -> 500 ms -> 5s ->10s. The maximum wait time per attempt before retry is limited to 10 seconds. Please note that if the overall request time exceeds the configured absolute_timeout it takes precedence, emits an error and prevents further retries. While the configured timeout value determines how long HTTP responses are allowed to take before an error is emitted.	4
no_trace_headers=1	Set to 1 to not add X-Dovecot-User or X-Dovecot-Session headers to HTTP request. These headers are useful to correlate object storage requests to App Suite/Dovecot sessions. If not doing correlations via log aggregation, this is safe to disable.	0
read_max_retries=<n>	Max number of HTTP request retries for read actions.	<max_retries> parameter
read_timeout=<time_msecs>	Timeout for a receiving read HTTP response.	<timeout> parameter
reason_header_max_length=<n>	Maximum length for X-Dovecot-Reason HTTP header. If header is present, it contains information why obox operation is being done.	0
slow_warn=<time_msecs>	Log a warning about any HTTP request that takes longer than this time.	5s
timeout=<time_msecs>	Default timeout for HTTP responses, unless overwritten by other parameters.	10s
write_max_retries=<n>	Max number of HTTP request retries for write actions.	<max_retries> parameter
write_timeout=<time_msecs>	Timeout for a write HTTP response.	<timeout> parameter

Example Configuration

WARNING

All text indicated by {{VARIABLE NAME}} in the examples below MUST be replaced with your local configuration value(s).

Note

Dictmap must also be configured to use this storage driver.

mail_location = obox:%u:INDEX=~/:CONTROL=~/

plugin {
  # SPROXYD_SCALITY_URL: The URL to the sproxyd interface.

  # Examples use Class of Service (CoS) 2 for mail, CoS 1 for FTS;
  # data compressed with zstd (level 3)

  # Without lazy_expunge plugin:
  obox_fs = fscache:512M:/var/cache/mails/%4Nu:compress:zstd:3:dictmap:proxy:dict-async:cassandra ; sproxyd:http://{{SPROXYD_SCALITY_URL}}/?class=2&reason_header_max_length=200 ; refcounting-table:lockdir=/tmp:bucket-size=10000:bucket-cache=%h/buckets.cache:nlinks-limit=3:delete-timestamp=+10s:bucket-deleted-days=11

  # With lazy_expunge plugin:
  #obox_fs = fscache:512M:/var/cache/mails/%4Nu:compress:zstd:3:dictmap:proxy:dict-async:cassandra ; sproxyd:http://{{SPROXYD_SCALITY_URL}}/?class=2&reason_header_max_length=200 ; refcounting-table:bucket-size=10000:bucket-cache=%h/buckets.cache:nlinks-limit=3:delete-timestamp=+10s:bucket-deleted-days=11

  obox_index_fs = compress:zstd:3:dictmap:proxy:dict-async:cassandra ; sproxyd:http://{{SPROXYD_SCALITY_URL}}/?class=2&reason_header_max_length=200 ; diff-table
  fts_dovecot_fs = fts-cache:fscache:512M:/var/cache/fts/%4Nu:compress:zstd:3:dictmap:proxy:dict-async:cassandra ; sproxyd:http://{{SPROXYD_SCALITY_URL}}/?class=1&reason_header_max_length=200 ; dict-prefix=%u/fts/
}

sproxyd API

See Scality sproxyd.

sproxyd Object ID Format

Dovecot uses its own Scality key format, which encodes the object type also to the key itself.

Key Format (160 bits)

Size	Description
152 bits	Entropy
4 bits	Class
4 bits	Replica

"Entropy" Component

Definition of entropy component of Scality's Universal Key Scheme (UKS):

Size	Description
24 bits	Dispersion (important)
64 bits	Object ID
32 bits	Volume ID
8 bits	Service ID (important)
24 bits	App-specific

So if we use 128 bit MD5 of the GUID as our unique key, this means we can do:

Size	Description
8 bits	Service ID = FS_SPROXYD_SERVICE_ID_DOVECOT = 0x83
8 bits	MD5 part suffix
8 bits	Object type (see below)
8 bits	Unused (0x00)
4 bits	Class ID (configurable)
4 bits	Replica ID (0x00)

Where "object type" is:

Internal Type	Value
FS_SPROXYD_OBJECT_TYPE_UNKNOWN	0x00
FS_SPROXYD_OBJECT_TYPE_MAIL	0x01
FS_SPROXYD_OBJECT_TYPE_OLD_FTS_INDEX	0x02
FS_SPROXYD_OBJECT_TYPE_FTS_INDEX	0x03
FS_SPROXYD_OBJECT_TYPE_USER_INDEX_SELF	0x08 | 0x01
FS_SPROXYD_OBJECT_TYPE_USER_INDEX_BASE	0x08 | 0x02
FS_SPROXYD_OBJECT_TYPE_USER_INDEX_DIFF	0x08 | 0x03
FS_SPROXYD_OBJECT_TYPE_BOX_INDEX_SELF	0x08 | 0x04 | 0x01
FS_SPROXYD_OBJECT_TYPE_BOX_INDEX_BASE	0x08 | 0x04 | 0x02
FS_SPROXYD_OBJECT_TYPE_BOX_INDEX_DIFF	0x08 | 0x04 | 0x03

For listing index objects, these can be helpful masks:

Internal Type	Value
FS_SPROXYD_OBJECT_TYPE_BIT_INDEX	0x08
FS_SPROXYD_OBJECT_TYPE_BIT_BOX_INDEX	0x04
FS_SPROXYD_OBJECT_TYPE_BIT_DIFF_OR_SELF	0x01

Utility Script

The scality-keys(1) script installed with the obox package takes a 160bit hex-encoded sproxyd Scality ID as input and outputs the object type.

Debug Headers

Dovecot sends the following HTTP headers towards storage. They should be logged for troubleshooting purposes:

• X-Dovecot-Username
• X-Dovecot-Session-Id
• X-Dovecot-Reason

Storage-Side Object Metadata

When saving data to object storage, Dovecot stores metadata associated with each blob for data recovery purposes.

This data is written to the HTTP endpoint by adding Dovecot metadata headers to the request. When retrieving a message from object storage, this data is returned in the received headers (only parsed by Dovecot if needed).

For S3, the header names are: X-Object-meta-dovecot-<key>.

Email Objects

Key	Description	Max Length (in bytes)	Other Info
fname	Dovecot filename	N/A (installation dependent; username component to naming)
guid	Message GUID	32
origbox	Folder GUID of first folder where stored	32	Copying does not update
pop3order	POP3 message order	10	Only if needed by migration
pop3uidl	POP3 UIDL	N/A (depends on source installation)	Only if message was migrated
received	Received data	20 (in theory; rarely more than 10)	UNIX timestamp format
saved	Saved data	20 (in theory; rarely more than 10)	UNIX timestamp format
size	Message size	20 (in theory; rarely more than 10)	Size in bytes
username	Dovecot unique username	N/A (installation dependent)

Index Objects

Key	Description	Max Length (in bytes)	Other Info
fname	Dovecot filename	N/A (installation dependent; username component to naming)
mailbox-guid	Mailbox GUID the index refers to	32
size	Message size	20 (in theory; rarely more than 10)	Size in bytes
username	Dovecot unique username	N/A (installation dependent)

FTS Index Objects

Key	Description	Max Length (in bytes)	Other Info
fname	Dovecot filename	N/A (installation dependent; username component to naming)
username	Dovecot unique username	N/A (installation dependent)