Obox Storage Support: Scality (sproxyd)

Configuration

Authentication

HTTP Basic authentication can be configured as part of the fs_sproxyd_url. URL escaping is used, so if password is foo/bar the URL is http://user:foo%2fbar@example.com/.

obox {
  fs sproxyd {
    fs_sproxyd_url = https://user:foo%2fbar@example.com/ # password is foo/bar
  }
}

Example Configuration

WARNING

All text indicated by {{VARIABLE NAME}} in the examples below MUST be replaced with your local configuration value(s).

TIP

Dictmap must also be configured to use this storage driver.

mail_driver = obox
mail_path = %{user}

# SPROXYD_SCALITY_URL: The URL to the sproxyd interface.
fs_sproxyd_url = http://{{SPROXYD_SCALITY_URL}}
fs_sproxyd_class = 2
fs_http_reason_header_max_length = 200
fs_compress_write_method = zstd
obox {
  fs fscache {
    size = 512M
    path = /var/cache/mails/%{user | sha1 % 4}
  }
  fs dictmap {
    dict proxy {
      name = cassandra
      socket_path = dict-async
    }
    #lock_path = /tmp # Set only without lazy_expunge plugin
  }
  fs sproxyd {
  }
}
metacache {
  fs compress {
  }
  fs dictmap {
    dict proxy {
      name = cassandra
      socket_path = dict-async
    }
  }
  fs sproxyd {
  }
}
fts dovecot {
  fs fts-cache {
  }
  fs fscache {
    size = 512M
    path = /var/cache/fts/%{user | sha1 % 4}
  }
  fs compress {
  }
  fs dictmap {
    dict proxy {
      name = cassandra
      socket_path = dict-async
    }
  }
  fs sproxyd {
    # It may be OK to store fewer copies of FTS indexes, because they can
    # be rebuilt if they become completely lost.
    class = 1
  }
}

Settings

fs_sproxyd

Value	Named Filter
See Also	Scality (sproxyd)

Filter for sproxyd-specific settings.

fs_sproxyd_access_by_path

Default	no
Value	boolean
Advanced Setting; this should not normally be changed.

Objects are accessed by the path instead of by the object ID. Scality sproxyd internally converts the paths into object IDs.

fs_sproxyd_avoid_423_timeout

Default	[None]
Value	unsigned integer
Advanced Setting; this should not normally be changed.

Delay DELETE requests if the same object ID has been GET/HEAD/PUT by the same process within Millisecond Time. This is intended to reduce 423 Locked sent by Scality.

When 0, no delay is added. Only use this setting, if it can be seen to bring a benefit. Careful investigation of current error-rates and consideration of the overall throughput of the platform are recommended before using it.

fs_sproxyd_class

Default	2
Value	unsigned integer

Scality Class of Service. 2 means that the objects are written to the Scality RING 3 times in total. This is generally the minimum allowable redundancy for mail and index objects.

FTS data is more easily reproducible, so losing those indexes is not as critical; Class of Service 1 may be appropriate based on customer requirements.

fs_sproxyd_url

Default	[None]
Value	string

URL for accessing the sproxyd storage.

Default HTTP Settings

sproxyd overrides some of the default HTTP client settings:

• http_client_max_idle_time = 1s
• http_client_max_parallel_connections = 10
• http_client_max_connect_attempts = 3
• http_client_request_max_redirects = 2
• http_client_request_max_attempts = 5
• http_client_connect_backoff_max_time = 1s
• http_client_user_agent = Dovecot/VERSION
• http_client_connect_timeout = 5s
• http_client_request_timeout = 10s

You can override these and any other HTTP client or SSL settings by placing them inside fs_sproxyd named filter.

sproxyd API

See Scality sproxyd.

sproxyd Object ID Format

Dovecot uses its own Scality key format, which encodes the object type also to the key itself.

Key Format (160 bits)

Size	Description
152 bits	Entropy
4 bits	Class
4 bits	Replica

"Entropy" Component

Definition of entropy component of Scality's Universal Key Scheme (UKS):

Size	Description
24 bits	Dispersion (important)
64 bits	Object ID
32 bits	Volume ID
8 bits	Service ID (important)
24 bits	App-specific

So if we use 128 bit MD5 of the GUID as our unique key, this means we can do:

Size	Description
8 bits	Service ID = FS_SPROXYD_SERVICE_ID_DOVECOT = 0x83
8 bits	MD5 part suffix
8 bits	Object type (see below)
8 bits	Unused (0x00)
4 bits	Class ID (configurable)
4 bits	Replica ID (0x00)

Where "object type" is:

Internal Type	Value
FS_SPROXYD_OBJECT_TYPE_UNKNOWN	0x00
FS_SPROXYD_OBJECT_TYPE_MAIL	0x01
FS_SPROXYD_OBJECT_TYPE_OLD_FTS_INDEX	0x02
FS_SPROXYD_OBJECT_TYPE_FTS_INDEX	0x03
FS_SPROXYD_OBJECT_TYPE_USER_INDEX_SELF	0x08 | 0x01
FS_SPROXYD_OBJECT_TYPE_USER_INDEX_BASE	0x08 | 0x02
FS_SPROXYD_OBJECT_TYPE_USER_INDEX_DIFF	0x08 | 0x03
FS_SPROXYD_OBJECT_TYPE_BOX_INDEX_SELF	0x08 | 0x04 | 0x01
FS_SPROXYD_OBJECT_TYPE_BOX_INDEX_BASE	0x08 | 0x04 | 0x02
FS_SPROXYD_OBJECT_TYPE_BOX_INDEX_DIFF	0x08 | 0x04 | 0x03

For listing index objects, these can be helpful masks:

Internal Type	Value
FS_SPROXYD_OBJECT_TYPE_BIT_INDEX	0x08
FS_SPROXYD_OBJECT_TYPE_BIT_BOX_INDEX	0x04
FS_SPROXYD_OBJECT_TYPE_BIT_DIFF_OR_SELF	0x01

Utility Script

The scality-keys(1) script installed with the obox package takes a 160bit hex-encoded sproxyd Scality ID as input and outputs the object type.

Debug Headers

Dovecot sends the following HTTP headers towards storage. They should be logged for troubleshooting purposes:

• X-Dovecot-Username
• X-Dovecot-Session-Id
• X-Dovecot-Reason

Storage-Side Object Metadata

When saving data to object storage, Dovecot stores metadata associated with each blob for data recovery purposes.

This data is written to the HTTP endpoint by adding Dovecot metadata headers to the request. When retrieving a message from object storage, this data is returned in the received headers (only parsed by Dovecot if needed).

For S3, the header names are: X-Object-meta-dovecot-<key>.

Email Objects

Key	Description	Max Length (in bytes)	Other Info
fname	Dovecot filename	N/A (installation dependent; username component to naming)
guid	Message GUID	32
origbox	Folder GUID of first folder where stored	32	Copying does not update
pop3order	POP3 message order	10	Only if needed by migration
pop3uidl	POP3 UIDL	N/A (depends on source installation)	Only if message was migrated
received	Received data	20 (in theory; rarely more than 10)	UNIX timestamp format
saved	Saved data	20 (in theory; rarely more than 10)	UNIX timestamp format
size	Message size	20 (in theory; rarely more than 10)	Size in bytes
username	Dovecot unique username	N/A (installation dependent)

Index Objects

Key	Description	Max Length (in bytes)	Other Info
fname	Dovecot filename	N/A (installation dependent; username component to naming)
mailbox-guid	Mailbox GUID the index refers to	32
size	Message size	20 (in theory; rarely more than 10)	Size in bytes
username	Dovecot unique username	N/A (installation dependent)

FTS Index Objects

Key	Description	Max Length (in bytes)	Other Info
fname	Dovecot filename	N/A (installation dependent; username component to naming)
username	Dovecot unique username	N/A (installation dependent)