Obox Storage Support: S3 Compatible

NOTE

Dovecot Pro/obox only directly supports the S3 service as provided directly by Amazon Web Services (AWS).

Dovecot Pro has a lower-tier of SLA support for other "S3-compatible" systems, but it is the customer's responsibility to ensure that system adequately implements the same behavior as AWS's S3 service.

S3 is not defined as an official protocol or API. "S3-compatible" systems attempt to replicate AWS's service offering.

Dictmap

Palomar with S3 Compatible storage requires dictmap.

Managed services exist that provide the necessary CQL infrastructure on AWS, such as AWS Keyspaces, DataStax Astra DB, or ScyllaDB Cloud. OX does not support configuration or operation of these managed services, and cannot provide recommendations or operational advice.

Configuration

S3 Compatible storage uses the s3 scheme for configuration:

plugin {
  # Basic configuration:
  obox_fs = s3:https://ACCESSKEY:SECRET@s3.example.com/?parameters
}

URL Formatting

An HTTP URL to specify how the object storage is accessed.

The parameters are specified as URL-style parameters, such as http://url/?param1=value1&param2=value2.

URL escaping is used, so if password is foo/bar the URL is http://user:foo%2fbar@example.com/.

Additionally, because Dovecot expands %variables inside the plugin section, the % needs to be escaped. So the final string would be e.g.:

plugin {
  obox_fs = s3:https://user:foo%%2fbar@example.com/ # password is foo/bar
}

Configuration Parameters

Parameter	Description	Default
absolute_timeout=<time_msecs>	Maximum total time for an HTTP request to finish. Overrides all timeout configuration.	None
addhdr=<name>:<value>	Add the specified header to all HTTP requests.	None
addhdrvar=<name>:<variable>	Add the specified header to all HTTP requests and set the value to the expanded variables value.	None
auth_host=<value>	IAM hostname and port. The default is 169.254.169.254:80. Normally there is no reason to change this. This is mainly intended for testing.	None
auth_role=<value>	Perform AWS IAM lookup using this role name.	None
bucket=<n>	Used by some backends to specify the bucket to save the data into.	None
bulk_delete_limit=<n>	Number of deletes supported within the same bulk delete request. 0 disables bulk deletes.	1000
connect_timeout=<time_msecs>	Timeout for establishing a TCP connection.	<timeout> parameter
delete_max_retries=<n>	Max number of HTTP request retries for delete actions.	<max_retries> parameter
delete_timeout=<time_msecs>	Timeout for sending a delete HTTP response.	<timeout> parameter
loghdr=<name>	Headers with the given name in HTTP responses are logged as part of any error, debug or warning messages related to the HTTP request. These headers are also included in the http_request_finished event as fields prefixed with http_hdr_. Can be specified multiple times.	None
max_connect_retries=<n>	Number of connect retries.	2
max_retries=<n>	Max number of HTTP request retries. Retries happen for 5xx errors as well as for 423(locked) with sproxyd. There is a wait between attempting next retry. The initial retry is done after 50ms. The following retries are done after waiting ten times as long as the previous attempt, so 50ms -> 500 ms -> 5s ->10s. The maximum wait time per attempt before retry is limited to 10 seconds. Please note that if the overall request time exceeds the configured absolute_timeout it takes precedence, emits an error and prevents further retries. While the configured timeout value determines how long HTTP responses are allowed to take before an error is emitted.	4
no_trace_headers=1	Set to 1 to not add X-Dovecot-User or X-Dovecot-Session headers to HTTP request. These headers are useful to correlate object storage requests to App Suite/Dovecot sessions. If not doing correlations via log aggregation, this is safe to disable.	0
read_max_retries=<n>	Max number of HTTP request retries for read actions.	<max_retries> parameter
read_timeout=<time_msecs>	Timeout for a receiving read HTTP response.	<timeout> parameter
reason_header_max_length=<n>	Maximum length for X-Dovecot-Reason HTTP header. If header is present, it contains information why obox operation is being done.	0
region=<value>	Specify region name for AWS S3 bucket. When this is specified, Dovecot starts using v4 signatures. (The default is to use v2 signatures.)	None
slow_warn=<time_msecs>	Log a warning about any HTTP request that takes longer than this time.	5s
timeout=<time_msecs>	Default timeout for HTTP responses, unless overwritten by other parameters.	10s
write_max_retries=<n>	Max number of HTTP request retries for write actions.	<max_retries> parameter
write_timeout=<time_msecs>	Timeout for a write HTTP response.	<timeout> parameter

Bucket Name

There are two ways to specify the bucket name in the configuration.

Path Style

S3 requests' path begins with the bucket parameter. This is configured with the bucket parameter.

For example: https://s3.example.com/?bucket=BUCKETNAME will result in requests looking like https://s3.example.com/BUCKETNAME/object-path

Virtual-Hosted Style

The first subdomain in the URL specifies the bucket.

NOTE

AWS S3 supports only this style for new buckets.

For example: https://BUCKETNAME.s3.example.com

Bulk Deletes

The S3 schemes support bulk-delete requests.

The bulk-delete option is enabled by default to delete up to 1000 keys with one request.

To change this behavior refer to bulk_delete_limit.

To actually delete that many mails in a single request, you must also set obox_max_parallel_deletes:

obox_max_parallel_deletes = 1000

This value should be the same as bulk_delete_limit or lower.

S3 Example Configuration

WARNING

All text indicated by {{VARIABLE NAME}} in the examples below MUST be replaced with your local configuration value(s).

Note

Dictmap must also be configured to use this storage driver.

mail_location = obox:%u:INDEX=~/:CONTROL=~/

plugin {
  # ACCESSKEY: Access key used for storage access.
  # S3_STORAGE_URL: URL for the S3 storage.
  # SECRET: Secret used for storage access.

  # Examples use data compression with zstd (level 3)

  # Without lazy_expunge plugin:
  obox_fs = fscache:2G:/var/cache/mails/%4Nu:compress:zstd:3:dictmap:proxy:dict-async:cassandra ; s3:https://{{ACCESSKEY}}:{{SECRET}}@{{S3_STORAGE_URL}}/?bucket=mails&reason_header_max_length=200 ; refcounting-table:lockdir=/tmp:bucket-size=10000:bucket-cache=%h/buckets.cache:nlinks-limit=3:delete-timestamp=+10s:bucket-deleted-days=11:storage-objectid-prefix=%u/mail-storage/
  # With lazy_expunge plugin:
  #obox_fs = fscache:2G:/var/cache/mails/%4Nu:dictmap:proxy:dict-async:cassandra ; s3:https://{{ACCESSKEY}}:{{SECRET}}@{{S3_STORAGE_URL}}/?bucket=mails&reason_header_max_length=200 ; refcounting-table:bucket-size=10000:bucket-cache=%h/buckets.cache:nlinks-limit=3:delete-timestamp=+10s:bucket-deleted-days=11:storage-objectid-prefix=%u/mail-storage/

  obox_index_fs = compress:zstd:3:dictmap:proxy:dict-async:cassandra ; s3:https://{{ACCESSKEY}}:{{SECRET}}@{{S3_STORAGE_URL}}/?bucket=mails&reason_header_max_length=200 ; diff-table:storage-passthrough-paths=full
  fts_dovecot_fs = fts-cache:fscache:2G:/var/cache/fts/%4Nu:compress:zstd:3:dictmap:proxy:dict-async:cassandra ; s3:https://{{ACCESSKEY}}:{{SECRET}}@{{S3_STORAGE_URL}}/?bucket=mails&reason_header_max_length=200 ; dict-prefix=%u/fts/:storage-passthrough-paths=full

  obox_max_parallel_deletes = 1000
}

S3 API

S3 Compatible servers must match the API behavior of AWS S3 API.

Debug Headers

Dovecot sends the following HTTP headers towards storage. They should be logged for troubleshooting purposes:

• X-Dovecot-Username
• X-Dovecot-Session-Id
• X-Dovecot-Reason

Storage-Side Object Metadata

When saving data to object storage, Dovecot stores metadata associated with each blob for data recovery purposes.

This data is written to the HTTP endpoint by adding Dovecot metadata headers to the request. When retrieving a message from object storage, this data is returned in the received headers (only parsed by Dovecot if needed).

For S3, the header names are: x-amz-meta-dovecot-<key>.

Email Objects

Key	Description	Max Length (in bytes)	Other Info
fname	Dovecot filename	N/A (installation dependent; username component to naming)
guid	Message GUID	32
origbox	Folder GUID of first folder where stored	32	Copying does not update
pop3order	POP3 message order	10	Only if needed by migration
pop3uidl	POP3 UIDL	N/A (depends on source installation)	Only if message was migrated
received	Received data	20 (in theory; rarely more than 10)	UNIX timestamp format
saved	Saved data	20 (in theory; rarely more than 10)	UNIX timestamp format
size	Message size	20 (in theory; rarely more than 10)	Size in bytes
username	Dovecot unique username	N/A (installation dependent)

Index Objects

Key	Description	Max Length (in bytes)	Other Info
fname	Dovecot filename	N/A (installation dependent; username component to naming)
mailbox-guid	Mailbox GUID the index refers to	32
size	Message size	20 (in theory; rarely more than 10)	Size in bytes
username	Dovecot unique username	N/A (installation dependent)

FTS Index Objects

Key	Description	Max Length (in bytes)	Other Info
fname	Dovecot filename	N/A (installation dependent; username component to naming)
username	Dovecot unique username	N/A (installation dependent)