Intercept Driver: Utimaco

Driver name: utimaco

The Utimaco LI framework is based on surveillance events, rather than capturing full protocol logs. These events are always related to manipulations of an e-mail message; login attempts are not captured.

In the Dovecot context, messages can be delivered through LMTP, retrieved from the server using POP3/IMAP, or stored in a mailbox using IMAP (APPEND). Each of these scenarios is subject to Utimaco surveillance.

The Utimaco driver has an additional service called utimaco-x1. This service implements the Utimaco X1 surveillance management protocol. Using this service, Utimaco can configure which users are targeted for surveillance. Surveillance targets are stored in a MySQL surveillance database and are identified by a UUID identifier. This identifier is assigned by Dovecot and used by Utimaco to refer to targets in the X1 management protocol.

Once surveillance events are recorded, they are submitted using the Utimaco X3 handover protocol. Several event parameters are passed for each event and the target UUID is used to identify the user to which this event belongs.

Configuration Options

`intercept_api_utimaco_sql_query_surveillance`

Default	`SELECT uuid, UNIX_TIMESTAMP(created) AS created FROM targets WHERE userid_hash = '%{user \| md5 \| hexlify}'`
Value	string
Advanced Setting; this should not normally be changed.

The SQL query used to obtain surveillance information (from the Utimaco X1 service) about a particular e-mail address.

See intercept_utimaco_x1_sql_query_check_surveillance for variable substitution information.

`intercept_utimaco`

Value	Named Filter

Named filter for intercept Utimaco driver settings.

`intercept_utimaco_x1_sql_query_add_surveillance`

Default	`INSERT INTO targets (userid_hash, uuid, address) VALUES ('%{userid \| md5 \| hexlify}', '%{uuid}', '%{address}')`
Value	string
Advanced Setting; this should not normally be changed.

The SQL query used to add a new target to the Utimaco X1 surveillance database.

See intercept_utimaco_x1_sql_query_check_surveillance for variable substitution information.

`intercept_utimaco_x1_sql_query_cancel_surveillance`

Default	`DELETE FROM targets WHERE uuid = '%{uuid}'`
Value	string
Advanced Setting; this should not normally be changed.

The SQL query used to remove a target from the Utimaco X1 surveillance database.

See intercept_utimaco_x1_sql_query_check_surveillance for variable substitution information.

`intercept_utimaco_x1_sql_query_check_surveillance`

Default	`SELECT uuid FROM targets WHERE userid_hash = '%{userid \| md5 \| hexlify}`
Value	string
Advanced Setting; this should not normally be changed.

The SQL query used to check (from Utimaco X1 service) whether an e-mail address is already surveilled.

Variable substitutions supported:

Variable	Description
`%{userid}`	User ID
`%{uuid}`	UUID
`%{address}`	Address

`intercept_utimaco_x1_sql_query_get_surveillance_status`

Default	`SELECT uuid FROM targets WHERE uuid = '%{uuid}'`
Value	string
Advanced Setting; this should not normally be changed.

The SQL query used to check (from the Utimaco X1 surveillance database) whether an UUID is currently surveilled.

See intercept_utimaco_x1_sql_query_check_surveillance for variable substitution information.

`intercept_utimaco_x1_sql_query_get_user_data`

Default	`SELECT userid_hash, address FROM targets WHERE uuid = '%{uuid}'`
Value	string
Advanced Setting; this should not normally be changed.

The SQL query used to get information (from the Utimaco X1 surveillance database) about a surveilled user.

See intercept_utimaco_x1_sql_query_check_surveillance for variable substitution information.

`intercept_utimaco_x1_sql_query_list_surveillances`

Default	`SELECT uuid FROM targets`
Value	string
Advanced Setting; this should not normally be changed.

The SQL query used to list all items from the Utimaco X1 surveillance database.

See intercept_utimaco_x1_sql_query_check_surveillance for variable substitution information.

`intercept_utimaco_x1_testing`

Default	`no`
Value	boolean
Advanced Setting; this should not normally be changed.

Enable the Utimaco testing mode for the utimaco-x1 service, in which the use of TLS is not required.

`intercept_utimaco_x3_debug`

Default	`no`
Value	boolean
Advanced Setting; this should not normally be changed.

Enable debug logging for utimaco-x3 service.

`intercept_utimaco_x3_kafka_brokers`

Default	[None]
Value	string

A list of brokers that are used to establish an initial connection to the Kafka cluster. At least one is required when Kafka is used. It is not required to list all brokers in the cluster; the full list is established during protocol initialization.

Separate brokers with spaces, , or ; in the setting.

`intercept_utimaco_x3_kafka_consume_interval`

Default	[None]
Value	time

The maximum time a broker is requested to wait before responding to a consumer. If there are no messages queued, the broker will respond with an empty list of messages. Use this to keep the connection with the broker active.

`intercept_utimaco_x3_kafka_consume_max_size`

Default	`10M`
Value	size
Advanced Setting; this should not normally be changed.

Limit a batch of consumed messages to approximately this size.

`intercept_utimaco_x3_kafka_consume_min_size`

Default	[None]
Value	size
Advanced Setting; this should not normally be changed.

The broker will respond to a consumer when a batch of messages with this cumulative size is available.

`intercept_utimaco_x3_kafka_consumer_group`

Default	`utimaco-x3-consumers`
Value	string
Advanced Setting; this should not normally be changed.

The name of the Kafka consumer group used by the utimaco-x3-kafka consumer service. This is currently only used to record the current consumption offset on the Kafka cluster; it is currently not possible to have more than a single consumer!

`intercept_utimaco_x3_kafka_topic`

Default	[None]
Value	string

The name of the Kafka topic the event messages are submitted to. This setting determines whether kafka is used; if this is not configured, the system will attempt direct submission to the X3 endpoint through HTTP.

`intercept_utimaco_x3_rawlog_dir`

Default	[None]
Value	string
Advanced Setting; this should not normally be changed.

If this directory exists, and it is writable by the utimaco-x3 service, the messages exchanged between client and server are logged here. This is useful for debugging.

`intercept_utimaco_x3_testing`

Default	`no`
Value	boolean
Advanced Setting; this should not normally be changed.

Enable the Utimaco testing mode for the utimaco-x3 service, in which the use of TLS is not required.

`intercept_utimaco_x3_url`

Default	[None]
Value	URL

The HTTP URL of the Utimaco X3 receiver.

Example Configuration

dovecot.confpassdb/userdb

### Intercept Proxy (Utimaco)

intercept utimaco {
}

# Time zone used by the intercept proxy. The Utimaco X3 protocol uses time
# stamps without time zone, so this setting is likely necessary to configure
# the time zone that the receiving Utimaco LI provider expects.
#intercept_tz = :Europe/Berlin

service intercept-proxy {
  unix_listener intercept-proxy {
    group =
    mode = 0666
    user =
  }
}

### Intercept Plugins

protocol lmtp {
  # The lda_intercept plugin captures delivered messages.
  mail_plugins {
    lda_intercept = yes
  }
}
protocol lda {
  # The lda_intercept plugin captures delivered messages.
  mail_plugins {
    lda_intercept = yes
  }
}

protocol imap {
  # The mail_intercept plugin captures all saved messages. For IMAP, this
  # means that messages uploaded with the APPEND command are captured.
  mail_plugins {
    mail_intercept = yes
  }

  # The imap_intercept plugin captures the full protocol exchange between
  # client and server. The intercept-proxy uses this data stream to extract
  # messages or message fragments that are retrieved using the FETCH command.
  mail_plugins {
    imap_intercept = yes
  }
}

protocol pop3 {
  # The pop3_intercept plugin captures the full protocol exchange between
  # client and server. The intercept-proxy uses this data stream to extract
  # messages or message fragments that are retrieved using the RETR and TOP
  # commands.
  mail_plugins {
    pop3_intercept = yes
  }
}

### Intercept API

protocols {
  intercept-api = yes
}

service intercept-api {
  # Enable debug mode for the service
  #executable = intercept-api -D

  # Listen on TCP port 7172 with SSL
  inet_listener ssl {
    port = 7172
    ssl = yes
  }
}

### Utimaco X1

protocols {
  utimaco-x1 = yes
}

intercept_utimaco_x1 {
  sql_driver = mysql
  mysql /var/run/mysqld/mysqld.sock {
    user = utimaco-x1
    password = utimaco-x1-password
    dbname = li
  }
}

protocol utimaco-x1 {
  # SSL certificates for the X1 service
  ssl_server_cert = </etc/dovecot/dovecot-utimaco.crt
  ssl_server_key = </etc/dovecot/dovecot-utimaco.key

  # Clients must present a certificate signed by this CA.
  ssl_server_ca = </etc/dovecot/utimaco.crt
  ssl_server_request_client_cert = yes
}

service utimaco-x1 {
  # Enable debug mode for the service
  #executable = utimaco-x1 -D

  # Listen at TCP port 7171 with SSL
  inet_listener ssl {
    port = 7171
    ssl = yes
  }
}

### Utimaco X3

# Config if connecting directly to X3 endpoint
intercept_utimaco_x3_url = https://utimaco-x3-server.example.com/
intercept_api {
  sql_driver = mysql
  mysql /var/run/mysqld/mysqld.sock {
    user = intercept-api
    password = secret-api-password
    dbname = li
  }
}
 
# Config if connecting via Apache Kafka
intercept_utimaco_x3 {
  kafka_brokers = https://broker-1.example.com/ \
      https://kafka-2.example.com/
  kafka_topic = dovecot_intercept
}

#service utimaco-x3 {
  # TLS is required for the X3 service. Client certificates usually need to
  # be configured as well:
  #ssl_client_cert = </etc/dovecot/dovecot-utimaco.crt
  #ssl_client_key = </etc/dovecot/dovecot-utimaco.key
#}

# Example of "normal" passdb/userdb lookups that account for subsequent
# surveillance database lookup.
#
# In this example, user is authenticated via passwd-file driver

passdb passwd-file {
  path = /etc/dovecot/dovecot.passwd
}

userdb passwd-file {
  path = /etc/dovecot/dovecot.passwd

  # Override username with canonical username (in this case without domain)
  # This is important!
  fields {
    user = %{user | username}
  }

  # If normal user lookup succeeds, we continue with the surveillance database
  # lookup.
  result_success = continue-ok

  # Fail immediately in case of (internal) failure
  result_failure = return-fail
  result_internalfail = return-fail
}

# User is authenticated; now lookup in the surveillance DB.
# Sets the 'intercept' variable if user exists in DB and is active. Everything
# else (including user not found) results in 'intercept' variable marked
# as false/empty.
userdb sql {
  skip = notfound

  # The SQL driver of the Utimaco surveillance database.
  driver = mysql

  # The SQL options used to connect the database. This database user
  # needs only read access.
  mysql /var/run/mysqld/mysqld.sock {
    user = dc-auth
    password = secret-auth-password
    dbname = li
  }

  # The SQL query used to lookup the user in the surveillance database. This
  # assigns the userdb field 'intercept'. This is interpreted by Utimaco driver
  # as the user's UUID in the surveillance database.
  query = SELECT uuid AS intercept FROM targets WHERE userid_hash = '%{user | md5}'
}

```sql[SQL Schema]
-- Example MySQL database structure for Utimaco Lawful Intercept

-- Create and use li database

CREATE DATABASE li;
USE li;

-- Create users; Only utimaco-x1 has write access

CREATE USER 'utimaco-x1'@'localhost'
  IDENTIFIED BY 'secret-utimaco-x1-password';
CREATE USER 'dc-intercept'@'localhost'
  IDENTIFIED BY 'secret-intercept-password';
CREATE USER 'dc-auth'@'localhost'
  IDENTIFIED BY 'secret-auth-password';
CREATE USER 'intercept-api'@'localhost'
  IDENTIFIED BY 'secret-api-password';

GRANT ALL PRIVILEGES ON li.* TO 'utimaco-x1'@'localhost';
GRANT SELECT ON li.* TO 'dc-intercept'@'localhost';
GRANT SELECT ON li.* TO 'dc-auth'@'localhost';
GRANT SELECT ON li.* TO 'intercept-api'@'localhost';

FLUSH PRIVILEGES;

-- Create tables

-- Surveillance targets
CREATE TABLE targets (
  -- Universally unique identifier for this surveillance target
  uuid VARCHAR(38) PRIMARY KEY,
  -- MD5 hash of the targeted user identifier. For SHA256, use CHAR(64).
  userid_hash CHAR(32) UNIQUE KEY,
  -- Automatic timestamp for creation of this target record
  created TIMESTAMP DEFAULT CURRENT_TIMESTAMP,

  address VARCHAR(320)
);

Target Identification

Users are surveilled only when the auth userdb lookup yields a non-empty intercept field.

For the Utimaco framework, this intercept field contains the user's UUID when the user is a surveillance target. The interception plugins pass the UUID along with the collected data to the intercept-proxy in the intercept-id session field.

The Utimaco driver needs the UUID in the X3 event submission, so passing it from the auth userdb lookup to the plugins and then to to the intercept-proxy prevents the need for doing an additional database lookup.

The auth service needs to be configured with an additional userdb lookup that adds the "intercept" field when surveillance is active for the user in question.

If an userdb is used solely to obtain intercept status for a user, the Dovecot authentication configuration MUST be configured so that other userdb lookups are also performed.

Interception Plugins

Required

Utimaco requires the mail-intercept plugin.

For IMAP and POP3, the Utimaco driver parses protocol messages to extract message retrieval events (POP3 RETR/TOP and IMAP FETCH). The pop3-intercept plugin and imap-intercept plugin plugins are used to capture the protocol messages and send them to the intercept-proxy. That is where the utimaco driver does the actual parsing.

In contrast, messages saved using IMAP APPEND are not extracted from the protocol flow between IMAP client and server. This is due to the fact that this is problematic when the IMAP CATENATE extension is used in combination with URLs. The intercept proxy cannot directly see the content that these URLs point to, making it difficult to capture the complete message. Therefore, messages that are saved using IMAP APPEND are captured using the mail-intercept plugin plugin. This plugin captures any saved message and sends it to the intercept plugin.

Messages delivered through LMTP are captured using the lda-intercept plugin plugin. This plugin captures any delivered message and sends it to the intercept plugin. Sieve cannot interfere with the plugin's operation.

Utimaco X1

The Utimaco driver has an additional service called utimaco-x1. This service implements the Utimaco X1 surveillance management protocol. It uses a mysql database to record the active surveillance targets.

Utimaco X3 Kafka

The Utimaco driver has an additional service called utimaco-x3-kafka. This service acts as a consumer for X3 event messages queued in Apache Kafka.

It passes the consumed messages on to the X3 endpoint at Utimaco through HTTP. Once accepted, the offset in Kafka is updated to point to the next event.

Kafka is mainly used to prevent the loss of event messages when the X3 endpoint at Utimaco is temporarily unavailable. It can also be used as a means to consolidate event messages from many backends and reliably send them to the X3 endpoint from a single IP.

X3 Message Queueing

Normally, Utimaco surveillance messages are directly submitted to the X3 protocol endpoint using a HTTP connection. When the X3 endpoint is not available, events will likely get lost. To prevent this, event messages can be queued through a local Apache Kafka cluster.

The intercept-proxy submits the events to Kafka, while a special utimaco-x3-kafka service retrieves the events from the Kafka cluster for the final submission to the X3 endpoint.

The utimaco-x3-kafka service has no listeners. It just runs a Kafka and an HTTP client. Events are consumed from the Kafka cluster and passed to the X3 endpoint through HTTP.

Implementation Status

Kafka supports partitioning messages within a topic, to spread the load of a topic over multiple brokers in the Kafka cluster. This is currently not implemented. Partition 0 is always used.

The Kafka client does not currently support the full feature set for Kafka consumers. This means that there can currently be only one utimaco-x3-kafka service consuming messages from Kafka.

Intercept API

The Utimaco driver provides a simple API in the intercept-api service. This API allows querying whether a particular user is surveilled. If the answer is positive, the target's UUID is returned along with the Utimaco X3 endpoint where surveillance events are to be submitted.

This API is needed for the external component that captures e-mail events from within the SMTP service.

See management API for details of the API.

Databases

Mechanisms

Users

Extensions

Intercept Driver: Utimaco ​

Configuration Options ​

intercept_api_utimaco_sql_query_surveillance

intercept_utimaco

intercept_utimaco_x1_sql_query_add_surveillance

intercept_utimaco_x1_sql_query_cancel_surveillance

intercept_utimaco_x1_sql_query_check_surveillance

intercept_utimaco_x1_sql_query_get_surveillance_status

intercept_utimaco_x1_sql_query_get_user_data

intercept_utimaco_x1_sql_query_list_surveillances

intercept_utimaco_x1_testing

intercept_utimaco_x3_debug

intercept_utimaco_x3_kafka_brokers

intercept_utimaco_x3_kafka_consume_interval

intercept_utimaco_x3_kafka_consume_max_size

intercept_utimaco_x3_kafka_consume_min_size

intercept_utimaco_x3_kafka_consumer_group

intercept_utimaco_x3_kafka_topic

intercept_utimaco_x3_rawlog_dir

intercept_utimaco_x3_testing

intercept_utimaco_x3_url

Example Configuration ​

Target Identification ​

Interception Plugins ​

Utimaco X1 ​

Utimaco X3 Kafka ​

X3 Message Queueing ​

Implementation Status ​

Intercept API ​

Intercept Driver: Utimaco

Configuration Options

`intercept_api_utimaco_sql_query_surveillance`

`intercept_utimaco`

`intercept_utimaco_x1_sql_query_add_surveillance`

`intercept_utimaco_x1_sql_query_cancel_surveillance`

`intercept_utimaco_x1_sql_query_check_surveillance`

`intercept_utimaco_x1_sql_query_get_surveillance_status`

`intercept_utimaco_x1_sql_query_get_user_data`

`intercept_utimaco_x1_sql_query_list_surveillances`

`intercept_utimaco_x1_testing`

`intercept_utimaco_x3_debug`

`intercept_utimaco_x3_kafka_brokers`

`intercept_utimaco_x3_kafka_consume_interval`

`intercept_utimaco_x3_kafka_consume_max_size`

`intercept_utimaco_x3_kafka_consume_min_size`

`intercept_utimaco_x3_kafka_consumer_group`

`intercept_utimaco_x3_kafka_topic`

`intercept_utimaco_x3_rawlog_dir`

`intercept_utimaco_x3_testing`

`intercept_utimaco_x3_url`

Example Configuration

Target Identification

Interception Plugins

Utimaco X1

Utimaco X3 Kafka

X3 Message Queueing

Implementation Status

Intercept API