Introduction
Crymap is an IMAP and SMTP server implementation for FreeBSD and Linux with a strong focus on security and simplicity of administration. Its spotlight feature is transparent encryption of data at rest — it is not possible to read any user’s mail without knowing their password, while a regular IMAP experience is provided and mail can be received while the user is offline.
If using the Crymap SMTP server, none of any user’s mail will ever be stored on disk in the clear (on your server, anyway), though note that Crymap SMTP has serious caveats.
Crymap supports both traditional UNIX-style deployments, where each user corresponds to a UNIX account and owns their own mail, and “black box” deployments, where users do not have shell access and all mail is owned by a single system UNIX account.
Features
- Fully compliant with the IMAP4rev1 and IMAP4rev2 specifications.
- Secure by default. IMAPS only.
- Minimal configuration.
- Messages and metadata transparently encrypted at rest.
- Automatic key rotation.
- Transparent file and over-the-wire compression.
- All normal mailboxes are “dual-use” (allow both messages and sub-mailboxes).
- Instant mail delivery notifications.
- QRESYNC support.
- “Special-use” mailbox support.
- A decent number of additional IMAP extensions.
- Supports messages with 8-bit and binary content.
- Mail delivery as an MDA.
- Mail delivery via SMTP or LMTP.
- Simple but secure sending of mail via SMTP with built-in DKIM support.
- Interoperates well with filesystem-based backup systems.
Status and Support
The author uses Crymap for all personal email. It is known to work well in this use case. But this is naturally a fairly small amount of experience; in particular, Crymap has only seen day-to-day use in conjunction with Thunderbird and FairEmail.
Crymap is currently maintained by its author alone. I am motivated to address bugs, but feature requests are unlikely to be accepted unless they offer substantial benefits to security or very common use cases. I will try to answer questions but cannot make commitments.
Caveats
-
If a password is forgotten, all data owned by that user is lost forever. There is no way for an administrator to reset a user’s password, as that would defeat Crymap’s purpose.
-
Crymap has no ability to integrate into the host authentication system. I.e., Crymap user accounts are fully independent of host user accounts in terms of password, enabled/disabled status, etc. This is because Crymap needs full control of the password change process for password changes to happen without destroying the user’s data. It would be technologically feasible to make, e.g., a PAM module that delegates to Crymap, but this is not implemented nor are there any plans to ever do so.
-
The maximum size of an email is currently hard-coded to 64MB.
-
Crymap’s outbound SMTP experience is unusual and somewhat cumbersome. If sending an email experiences a temporary failure, it cannot be automatically retried since all access to messages is cryptographically locked behind user authentication. Retrying must be done manually via an IMAP extension, which is currently only implemented by the Crymap CLI utility. For a more conventional experience, you can use something like OpenSMTPD to handle outbound messages instead.
How Crymap Encryption Works
Crymap uses a number of modern, well-understood cryptographic primitives to securely store email data.
Encrypting Message Data
Messages must be stored in a way that ensures two properties:
-
Only the owner can read mail.
-
Anyone with file system access (e.g., the mail daemon) can add mail to the user’s mailbox.
This is a natural fit for asymmetric encryption. Currently, Crymap exclusively uses 4096-bit RSA for this purpose.
Each account has, at any given time, a single active public key. When mail is received, an encryption key (the “session key”) for the message is randomly generated, and the message data is stored encrypted with that session key. The session key itself is first encrypted with the active public key. This lets the user (who has access to the corresponding private key) decrypt the session key and then decrypt the message content.
The message content itself is encrypted with 128-bit AES-GCM. Besides making it impossible for others to read the message, it also ensures that if the data is corrupted, Crymap will signal an error instead of returning the corrupt data.
A Crymap account typically has more than one RSA private key, though only one at a time corresponds to the active public key. By default, keys are rotated once per month. (The old keys are retained though, so old messages are still readable.) By default, there is also a distinction between “internal” keys, which never have their public key exposed and are used for all encryptions done by the user while logged in, and “external” keys, whose public keys are made available as the primary public key as each key is introduced.
RSA private keys are stored in standard PEM format with a passphrase. The passphrase is not the user’s passphrase, but one autogenerated from the user’s login data (see next section).
User Credentials
Handling of each user’s passphrase is performed with a combination of two cryptographic primitives:
-
The Argon2 password hashing function, which makes password brute-force attacks extremely costly.
-
The KMAC function (essentially an HMAC based on Keccak, i.e., SHA-3), which is used to generate secondary keys from an input secret.
The user configuration stores three pieces of information about the user’s credentials:
- A password salt
- A password hash
- The “master key XOR”
When the user inputs their password, it is first hashed using Argon2 with the given salt, giving what we’ll call the “argon hash”. A KMAC value is generated from the argon hash. If that KMAC is not equal to the user’s password hash, we know the password is wrong and can reject the log in. Otherwise, we know the password is right, and take a different KMAC value from the argon hash to produce an “intermediate hash”. Finally, every byte of the intermediate hash is XORed with the corresponding byte of the master key XOR to output the master key.
The “master key XOR” is functionally a one-time pad between the intermediate hash and the master key. Its significance is that it makes it possible for the user to change their password: there is no way to control what the intermediate hash of the new password will be, but whatever it is, a master key XOR can be chosen which still outputs the same master key for the new password as the previous set of parameters did for the old password.
The master key is the primary credential used for all further key derivation in Crymap. Currently, the only derived keys are the PEM passphrases for the RSA private keys, which are each generated from different KMACs of the master key.
Protection Crymap Provides
There are a number of different approaches one can take with encrypting mail. None is a perfect solution, and Crymap fills what is currently a unique niche.
Protection from… | Disk | End-to-end | PGPipe | Crymap |
---|---|---|---|---|
Theft of storage medium | Yes | Partial | Partial | Yes |
File exfiltration | No | Partial | Partial | Yes |
Metadata exposure | No | No | No | Partial |
Eavesdropping in-transit | No | Partial | No | No |
Connection compromise | No | Partial | Partial | No |
Disk encryption
Disk encryption is a simple, effective way of protecting data in case someone walks away with the physical storage medium. It does not protect against attackers gaining shell access to the server or any form of file exfiltration since all the encrypted files are available in cleartext to anyone who can access that part of the file system. Disk encryption also poses challenges with respect to booting the server up without human intervention.
-
Protects from theft of storage medium.
-
No protection from file exfiltration.
-
No protection of message metadata.
-
No protection from eavesdropping on message in-transit.
-
No protection from IMAP connection compromise.
End-to-End Encryption
PGP and S/MIME provide ways for a sender to encrypt a message so that only the receiver can read it. Unlike the other solutions in this list, the message is also secure in-transit. The only way to compromise email content is to compromise the end email client itself. Unfortunately, not many people send encrypted messages this way. End-to-end encryption also typically reduces the quality of the mail experience. There is no protection of message metadata which is outside the message itself. Finally, PGP fails to encrypt any of the message headers.
-
Protects from theft of storage medium, except for parts of the message that can’t be encrypted.
-
Protects from file exfiltration, except for parts of the message that can’t be encrypted.
-
No protection of message metadata.
-
Protects from eavesdropping on message in-transit, except for parts of the message that can’t be encrypted.
-
Protects from IMAP connection compromise, except for parts of the message that can’t be encrypted.
Better-late-than-never encryption
This means using a solution like PGPipe to apply PGP encryption to unencrypted messages right before delivered. This protects the message after it has been delivered, but comes with the other downsides of PGP.
-
Protects from theft of storage medium, except for parts of the message that can’t be encrypted.
-
Protects from file exfiltration, except for parts of the message that can’t be encrypted.
-
No protection of message metadata.
-
No protection from eavesdropping on message in-transit.
-
Protects from IMAP connection compromise, except for parts of the message that can’t be encrypted.
Crymap
Since Crymap encrypts all message data at rest and has per-user isolation, it provides full defence against theft of storage medium and file exfiltration. It is also able to protect most message metadata, such as message flags. An attacker with file system access can still use that access to form educated guesses about when certain messages were delivered and their approximate size. On the other hand, since Crymap transparently decrypts data before sending it over IMAP, compromising an IMAP connection reveals all data going over the wire.
-
Protects from theft of storage medium.
-
Protects from file exfiltration.
-
Partial protection of message metadata.
-
No protection from eavesdropping on message in-transit.
-
No protection from IMAP connection compromise.
Installation Guide
System Requirements
Crymap is designed to run on FreeBSD and Linux. It probably works on the other BSDs, but is not tested on them. Windows is not supported and most likely never will be since its filesystem is incompatible with Crymap’s requirements on many fronts.
The host file system must support symlinks and hard links. There must not be mount points inside any user directory. Remote file systems like NFS are supported if they provide sufficient consistency guarantees, but Crymap is not specifically designed to work with them, and IDLE notifications will only work properly if all Crymap instances for a given user are run on the same host.
When run on a single-CPU host or with appropriate tuning on a multi-CPU host, each Crymap instance typically only consumes a few MB of RAM. However, do be aware that Crymap is a one-process-per-connection system.
Crymap generally only holds at most a few dozen file handles open at any given time.
Installation
Crymap requires having the OpenSSL libraries installed. On Debian/Ubuntu, you
can get them with apt install openssl
. FreeBSD has the required libraries in
the base distribution. (There is a way to cause OpenSSL to be built along with
Crymap and statically linked, but this is not recommended since you won’t be
able to get OpenSSL updates with your OS and this process is not described in
this manual.)
Each [https://github.com/AltSysrq/crymap/releases/](release of Crymap) has
pre-built binaries for AMD64 Debian and FreeBSD. Simply download the
appropriate one of these and save it in a place you consider appropriate; for
example (#
means “as root”):
# cp crymap-$version-$os-amd64 /usr/local/bin/crymap
# chmod a=rx /usr/local/bin/crymap
If you do not want to use the pre-built binary or need to run on a different OS or architecture, you need to build Crymap from source. This requires having Rust installed, which you can get through rustup.
The easiest way to build Crymap is to get it with cargo
:
$ cargo install crymap
# cp ~yourusername/.cargo/bin/crymap /usr/local/bin/crymap
You can also clone the repository directly:
$ git clone git@github.com:altsysrq/crymap
$ cd crymap
$ cargo build --release
# cp target/release/crymap /usr/local/bin/crymap
Initial System Setup
First, make sure you have valid SSL certificates for your domain. A full explanation of how to do this is beyond the scope of this document, but Let’s Encrypt is a good place to start if you are new to this.
All files used by Crymap are reachable from (though not necessarily “in”) a
directory referred to as the “Crymap root”. By default, this is either
/etc/crymap
or /usr/local/etc/crymap
(whichever exists); you can use
something else, but if you do so, you must pass --root
to all crymap server
commands to tell Crymap where its data is. For this tutorial, we’ll use
/usr/local/etc/crymap
as the root.
Start by creating the Crymap root directory. The files we’re setting up below
shouldn’t be modified in normal operation, so they should be owned by root
.
# mkdir /usr/local/etc/crymap
Next, we create Crymap’s system configuration file, which is a file named
crymap.toml
under the Crymap root. A minimal configuration example is shown
below.
# /usr/local/etc/crymap/crymap.toml
[tls]
# The path to your X509 private key. The example here would be correct for
# a FreeBSD system at `example.org` using Let's Encrypt with the default
# configuration.
private_key = "/usr/local/etc/letsencrypt/live/example.org/privkey.pem"
# The path to the full X509 certificate chain.
certificate_chain = "/usr/local/etc/letsencrypt/live/example.org/fullchain.pem"
(Yes, that’s all there is to the required configuration.)
Before we go further, we need to decide what style of deployment we’re going to do. Crymap supports three general patterns:
-
UNIX-style deployment, in which each Crymap user corresponds to a UNIX user on the host system and owns their Crymap files.
-
Simple black box deployment, where Crymap users are unrelated to UNIX users, and all Crymap data is owned by a dedicated UNIX account. Crymap is always run under that UNIX account.
-
Two-step black box deployment. As above, but Crymap is run as
root
and then allowed to drop privileges once it no longer needs them. This requires more work to set up but means that the dedicated Crymap user doesn’t need permission to read SSL certificates and the like. This chapter won’t talk about this approach; simply go through the setup for the simple black box deployment, then refer to the configuration reference to see what to change.
If doing a black box deployment, set up the dedicated Crymap user now. Below,
we’ll assume the user has name crymap
and group mail
, but of course that
will vary with your system.
The next step is to create the users
directory.
If doing a UNIX-style deployment, the user’s mail will typically be stored in
their home directory. This means that users
will not actually contain the
data and we can just make it a normal directory.
# mkdir /usr/local/etc/crymap/users
On a black-box deployment, the user data will end up under whatever directory
we create, and we don’t really want to be storing data under /etc
. Thus, we
will create the users
directory elsewhere and symlink it from the nominal
location. In this example, we store users under /srv/crymap-users
.
# mkdir -m 750 /srv/crymap-users
# chown crymap:mail /srv/crymap-users
# ln -s /srv/crymap-users /usr/local/etc/crymap/users
To see if your configuration so far is good, try running crymap serve serve-imaps
under the same UNIX account you intend using in the real
deployment. If you get the message stdin and stdout must not be a terminal
,
it means the configuration is valid and Crymap was ready to serve network
traffic.
We won’t create any users just yet.
Running Crymap in IMAPS mode
Crymap does not run as a daemon. Instead, it uses a one-process-per-connection
model in which some other process listens for connections and spawns Crymap
processes as connections arrive. There are a number of options here, including
xinetd and systemd. Here, we will use traditional inetd, available as the
package openbsd-inetd
on Debian and part of the base FreeBSD installation.
Arranging for Crymap to be run by inetd is just a matter of adding two lines to
/etc/inetd.conf
and then restarting inetd:
imaps stream tcp nowait root /usr/local/bin/crymap crymap server serve-imaps
imaps stream tcp6 nowait root /usr/local/bin/crymap crymap server serve-imaps
If setting up a simple black box deployment, replace root
with whatever user
you want to run Crymap as.
Note the second occurrence of crymap
in the command line is because inetd
requires the configuration to specify the usually implicit argv[0]
explicitly.
Do not add entries for the imap4
service. Crymap does not support the
obsolete cleartext+STARTTLS
mechanism
on the IMAP4 port 143.
To test whether this setup works, run crymap remote test --trace --host=localhost
. (You can of course run from another system, in which case
pass the server’s host name to --host
.) If successful, you should see some
IMAP protocol traces and receive a prompt for a password. Of course, since no
users have been created yet, it is not possible to log in, so simply cancel
once it gets that far.
Creating a User
User creation is done with the crymap server user add
command. By default, it
generates a random password for the new user; --prompt-password
can be given
to input a password on the terminal instead. Refer to the user
guide to see how to change the password after the account has
been created.
On a UNIX-style setup, this command should be run as root
, and you’ll usually
want to give the path to the user data explicitly.
# crymap server user add jsmith ~jsmith/crymap-mail
On a black box setup, this command should be run as the Crymap user, and if
your users
directory is symlinked to a location outside of /etc
, you can
leave the user directory off to simply store the user data inside users
.
$ crymap server user add jsmith
With that done, you should be able to test logging in as the new user:
$ crymap remote test --host=localhost --user=jsmith
It is also possible at this point to connect your favourite email application to Crymap, though you can’t receive email just yet.
Configuring Inbound Mail Delivery
There are three ways to set up inbound mail delivery.
-
SMTP. In this setup, Crymap receives messages directly from external servers. Having Crymap handle SMTP directly is the simplest setup and ensures no message data is written to disk in the clear, but is also the least flexible option, as Crymap won’t do anything with inbound messages but deliver them to the INBOX without flags.
-
LMTP. LMTP is a variant of SMTP which is suitable for the final step in the mail delivery process. Compared to having Crymap serve SMTP directly, LMTP is more complicated to set up as you need a third-party SMTP solution and may be less secure since most SMTP servers spool messages to disk in cleartext, but will give you whatever flexibility the SMTP solution offers. Compared to the UNIX MDA option below, LMTP is robust, can correctly handle binary payloads, and can be more efficient, but is inflexible in that all messages are delivered to INBOX and without flags.
-
UNIX MDA. This is more flexible than SMTP LMTP as it can be made to deliver to arbitrary mailboxes in a user’s account and to set flags on new messages. In traditional UNIX setups, users can also invoke Crymap in as an MDA from their
.forward
file. Its main disadvantages are that it is less robust against errors, and because UNIX MDAs are typically passed the message with all line endings converted to UNIX line endings, Crymap must convert the line endings back, which will destroy binary content. Like LMTP, this also comes with the disadvantages of needing an external SMTP solution.
SMTP
First, inbound SMTP requires a bit of extra configuration in crymap.toml
.
You’ll need to tell Crymap what its fully-qualified domain name is and what
domains it is expecting to serve.
The below example shows what you might add to a server that will be receiving
email from example.com
and example.net
.
[smtp]
# A DNS lookup for this name should return the IP address of the Crymap server.
host_name = "mx.example.net"
[smtp.domains."example.com"]
# No configuration here for now, we just need to make sure the section for
# "example.com" exists.
[smtp.domains."example.net"]
# No configuration here for now, we just need to make sure the section for
# "example.com" exists.
Next, you just need to arrange for Crymap to handle inbound SMTP connections. Here is an example inetd configuration for a UNIX-style deployment:
smtp stream tcp nowait root /usr/local/bin/crymap crymap server serve-smtpin
smtp stream tcp6 nowait root /usr/local/bin/crymap crymap server serve-smtpin
Notes:
-
Notice this uses
serve-smtpin
instead ofserve-imaps
at the end. -
As with IMAPS, for a simple black box deployment, you would configure it to run under the Crymap user and not
root
.
You can use socat
to see if SMTP is working:
$ socat STDIO TCP:localhost:25
220 mx.example.net crymap 2.0.0 ESMTP ready
^C
At this point, you should also be able to send email from external email providers to your server.
LMTP
Besides setting up crymap, this section also contains concrete examples of using OpenSMTPD; if you are using a different SMTP daemon, you will need to refer to its documentation.
First, we need to set something up to run Crymap in LMTP mode. This works essentially the same as how we set up IMAPS, except here we don’t want the server to be accessible to anyone in the world. There’s two ways to do that:
-
Configure inetd, etc, to only listen on localhost.
-
Use a UNIX socket. This is what we do in this example, since it is possible to set finer-grained permissions.
Here is an example inetd configuration for a UNIX-style deployment:
:root:wheel:666:/var/run/lmtp.sock stream tcp nowait root /usr/local/bin/crymap crymap server serve-lmtp
Notes:
-
Notice this uses
serve-lmtp
instead ofserve-imaps
at the end. -
root:wheel
would be writtenroot:root
on typical Linux installations. -
As with IMAPS, for a simple black box deployment, you would configure it to run under the Crymap user and not
root
. -
This example allows anyone with shell access to open the LMTP socket and deliver mail (due to the
666
permission). To prevent that, you can give the socket a different owner and less permissive permissions. But to start with, it is probably best to keep it simple so there are fewer things to debug.
Crymap has a number of configurable options for LMTP. The defaults are fine for simple installations, but you may want to have a look at the configuration reference.
You can use socat
to see if LMTP is working:
$ socat STDIO UNIX:/var/run/lmtp.sock
220 yourhostname crymap 2.0.0 LMTP ready
^C
Finally, you need to configure your SMTP server to deliver mail through LMTP.
For OpenSMTPD, your smtpd.conf
file might contain something like this:
action "local_mail" lmtp "/var/run/lmtp.sock"
match from any for domain "lin.gl" action "local_mail"
match for local action "local_mail"
After that, all that’s left to test is to send the user an email and see if it shows up!
Unix MDA
There isn’t anything Crymap-specific to set up for this use case. You will need
to find out how to get your SMTP solution to run a UNIX MDA command line, which
in most cases would simply be crymap server deliver
. Refer to
crymap server deliver --help
for additional options.
Outbound SMTP
If you want to use a third-party solution for outbound mail, there is nothing to configure which is Crymap-specific. Third-party solutions are generally more user-friendly, more battle-tested, and more flexible, but are less secure since they need to spool messages onto disk in cleartext or another way that allows passive recovery of the message content.
Crymap does have outbound SMTP support. Its advantages are simple configuration, built-in DKIM support, unification of the authentication system with IMAP, and that messages do not get spooled to disk in the clear. However, it has a major downside: Retrying failed messages is manual, and currently can only be done with the Crymap command-line application.
If you want to use Crymap’s outbound SMTP support, you will first need to add
the SMTP configuration to crymap.toml
if you haven’t done this already. Refer
to the SMTP section for an example.
If you have existing DKIM keys you wish to continue using, you can add these to
the domain-specific sections in crymap.toml
. Search for “DKIM” in the
Configuration Guide for details on what to add there.
You can use the openssl
and base64
command-line utilities to convert the
private keys you already have into the required format.
If you do not have existing DKIM keys, the next step will help you generate some.
You can use crymap server smtp-out-sanity-check
on your server to check your
configuration, including external DNS. This tool will generate DKIM keys for
you if you have none, and will provide additional suggestions to make it more
likely that major email providers will accept your email.
Finally, you need to arrange for Crymap to run SMTP submission to receive outbound mail from your users. There are two supported options here:
-
SMTP+TLS on port 465, called variously
submissions
,smtps
, orssmtp
in/etc/services
, is preferred. Use only this one if you can. This is theserve-smtpssub
subcommand (note the double “s”). -
SMTP submission on port 587, usually called
submission
in/etc/services
. This variant relies on the client performingSTARTTLS
and may be vulnerable to the “strip TLS” attack depending on the client. This is theserve-smtpsub
(note no double “s”).
Here is an example inetd configuration for a UNIX-style deployment with both protocols:
# The preferred "implicit TLS" variant
submissions stream tcp nowait root /usr/local/bin/crymap crymap server serve-smtpssub
submissions stream tcp6 nowait root /usr/local/bin/crymap crymap server serve-smtpssub
# The legacy STARTTLS variant
submission stream tcp nowait root /usr/local/bin/crymap crymap server serve-smtpsub
submission stream tcp6 nowait root /usr/local/bin/crymap crymap server serve-smtpsub
Notes:
-
Sotice this uses
serve-smtpssub
andserve-smtpsub
instead ofserve-imaps
at the end. -
As with IMAPS, for a simple black box deployment, you would configure it to run under the Crymap user and not
root
.
You can use socat
to see if it is working:
# Implicit TLS protocol
# You could also use your server's fully-qualified domain name instead of
# "localhost" and then exclude ",verify=0" instead.
$ socat STDIO SSL:localhost:465,verify=0
220 mx.example.net crymap 2.0.0 ESMTPS ready
^C
# Legacy STARTTLS protocol
$ socat STDIO TCP:localhost:587
220 mx.example.net crymap 2.0.0 ESMTP ready
^C
You are now ready to test sending mail. You should start by emailing yourself so that bad configuration does not impact your reputation on the big providers, then move on to various third parties.
If sending a message fails at the SMTP level, the failure receipt will be delivered to your account’s inbox, which you can use to debug why the remote server rejected your message.
Troubleshooting
By default, Crymap logs to syslog under the “mail” utility. When Crymap is not
run from a terminal, this means its logs should end up someplace like
/var/log/maillog
or /var/log/mail.log
.
In certain cases of severe misconfiguration (for example, by passing invalid
parameters in inetd.conf
), Crymap may instead print an error on standard
error and exit. Where these messages end up depends on what you are using to
run Crymap. The output might be in a system log like /var/log/messages
, or it
could be getting sent over the socket. To check for the latter case, you can
run a command like socat STDIO TCP:localhost:993
to see if anything shows up.
Administration Guide
These sections cover (hopefully) everything you need to know about operating and maintaining a Crymap installation beyond the initial setup.
Configuration Reference
crymap.toml
Below is an example crymap.toml
with every option explicitly set to the
default (or an example value if there is no default) and comments explaining
what each one does.
[tls]
# The path to your X509 private key.
private_key = "<no default>"
# The path to the full X509 certificate chain.
certificate_chain = "<no default>"
# Additional identification information to send to clients.
# This is a free-form map. Refer to RFC 2971 § 3.3 to see what standard
# identification names exist. You do not need to set all the values.
# `support_url` is probably the most important one since some mail clients
# can use it to help the user get assistance.
# Underscores in key names are replaced with hyphens.
# The contents of this section is examples and not defaults, as the default
# configuration is empty.
[identification]
vendor = "Example Company"
support_url = "mailto:it@example.com"
address = "1313 Dead End Dr"
# The [security] section applies any time any of the `crymap server ...`
# commands is run.
[security]
# If this is set to true and Crymap is run as `root`, it will chroot into the
# `users` path under the Crymap root once it has accessed all system files it needs
# which are outside that directory.
#
# This can be set for any style of deployment where Crymap is started as
# `root`, but note that if this is set, user directories inside `users` may not
# be absolute symlinks or symlinks to anywhere outside `users`.
chroot_system = false
# If this is set to a non-empty value and Crymap is run as `root`, it will drop
# privileges to this UNIX user once it has accessed all system files it needs
# which are outside the Crymap root. This occurs before any interaction with the
# incoming connection occurs. This makes it possible to run Crymap in a
# configuration where it does not normally have access to SSL certificates, for
# example.
#
# This setting is the main difference between a "simple" black box deployment,
# where Crymap is started as the desired non-root user, and a "two-step" black
# box deployment, where Crymap is started as root and then drops to the user
# given here.
system_user = ""
# The [smtp] section applies when Crymap is run with `crymap server serve-lmtp`,
# `crymap server serve-smtpin`, `crymap server serve-smtpsub`, and
# `crymap server serve-smtpssub`.
#
# In Crymap 1.x, this section was called `[lmtp]`, and that name is still
# supported for backwards-compatibility.
[smtp]
# If non-empty, Crymap will report this value as its hostname.
# By default, Crymap reports the system host name.
# Explicit configuration of this value is required for outbound SMTP support
# as a fully-qualified domain is required in that case.
host_name = ""
# By default, Crymap will strip everything after the `@` in destination
# addresses to determine the name of the Crymap user (so an email addressed
# to "foo@bar.com" gets delivered to Crymap user "foo"). Setting this to true
# suppresses this behaviour. Note that your SMTP daemon may also do this
# stripping, in which case you must configure both Crymap and the SMTP daemon
# to retain the domain.
#
# Set to true if your users log in as `user@domain.com` instead of `user`.
keep_recipient_domain = false
# By default, Crymap will make the recipient user name lower case, remove all
# periods, and strip everything including and after the first `+`. (If
# `keep_recipient_domain` is `true`, the `@` and domain part are not affected
# by these rules, but the local part is.) Setting this to true prevents all
# these normalisations, which may be useful if you want something different to
# happen. Note that this means that your SMTP daemon must make the
# normalisations you want. Most importantly, enabling this option will make
# Crymap mail delivery CASE SENSITIVE TO USER NAMES.
#
# You probably shouldn't set this to true if Crymap is fronting SMTP itself.
verbatim_user_names = false
# By default, Crymap evaluates DMARC for inbound SMTP but does not take any
# action on DMARC failures.
# If set to true, Crymap will reject inbound SMTP transactions if the DMARC
# evaluation fails and the DMARC record requests to reject such failures.
#
# This has no effect on LMTP or SMTP submission.
reject_dmarc_failures = false
# If enabled, receipts produced for outbound SMTP transactions will include
# very verbose details about TLS handshakes.
verbose_outbound_tls = false
# Each entry under `smtp.domains` describes an SMTP domain the Crymap server
# will support. You only need to configure this if Crymap is handling SMTP
# itself.
# Simply defining a table, even if empty, is sufficient to make Crymap consider
# the domain usable.
[smtp.domains."example.com"]
# Each value under `dkim` defines a DKIM key which will be used to sign
# messages originating from this domain. They have no effect on inbound
# messages.
#
# The part after `dkim.` is the DKIM selector. The value is either `rsa:DATA`,
# where DATA is a base64-encoded RSA private key in DER format, or
# `ed25519:DATA`, where DATA is a base64-encoded ED25519 private key in raw
# format.
#
# If there are multiple keys for one domain, they will all be used to sign the
# message, in lexicographical order. Note that Microsoft Exchange only examines
# the first DKIM header on a message, and fails the message if it does not know
# about the signature algorithm. As of 2024-05, it does not understand ED25519,
# so you should ensure that the first key (in lexicographical order, not
# necessarily the written in the configuration) is a reasonably-sized RSA key
# if you care about talking to people on Exchange/Outlook.com/Hotmail/etc.
dkim.selector1 = "rsa:MIQU2whmZwg<VERY LONG STRING...>"
dkim.selector2 = "ed25519:13SSM<LONG STRING...>"
[diagnostic]
# If set, redirect standard error to this file on startup.
#
# This is applied before any part of the security configuration is
# applied and before any communication with the remote host.
#
# This is useful if `inetd` (or equivalent) or your MTA runs Crymap such
# that standard error goes to a less useful place, such as to the remote
# host. If anything actually ends up in this file, it represents a bug in
# Crymap, as actual errors should go through the logging system.
stderr = null
Logging
By default, Crymap logs to syslog under the “mail” facility.
If you want to do something else, you can create a file named logging.toml
next to crymap.toml
. This file is a log4rs
configuration file. The exact format of this file is not extensively
documented, so if you want to do this, you’re unfortunately on your own for
now. Note that there is not currently a way to log to syslog and a separate
logging system simultaneously at this time.
Here is a basic example of a logging.toml
file:
[appenders.file]
kind = "rolling_file"
path = "/var/log/crymap/crymap.log"
append = true
[appenders.file.policy.trigger]
kind = "size"
limit = "10 MB"
[appenders.file.policy.roller]
kind = "delete"
[appenders.file.encoder]
kind = "pattern"
pattern = "{d(%Y-%m-%dT%H:%M:%S)} [{l}][{t}] {m}{n}"
[root]
level = "info"
appenders = ["file"]
Managing Users
Crymap’s notion of a user
A “user” in Crymap simply refers to an entry within the users
directory. Each
entry may either be a directory, or be a symlink to a directory. Whichever it
is, that directory is the user data directory.
In traditional UNIX-style deployments, Crymap uses the owner of the user data directory to determine which UNIX account to assume for operations on that user. In black box deployments, the exact ownership of the user data directory is less important, but it is still necessarily something that the Crymap user has access to and which doesn’t let unauthorised users access it.
Creating users
Creation of a user is (internally) a non-trivial operation since it needs to generate a master key for the user and make it derivable from the user’s initial password. The process also must set up the user’s basic directory structure and mailboxes, most importantly INBOX.
User creation is done through the crymap user add
command. This command
should be run as root
for traditional UNIX-style deployments and as the
Crymap user for black box deployments.
The first argument of the command is the name of the user to create, e.g.,
jsmith
. If the command is run as root
, Crymap will by default assume that
that name also refers to the name of the UNIX account that will own the mail
data, and will fail if no such account exists. The --uid
option can be passed
to provide the UID that should own the user data directory.
The optional second argument of the command gives the path to the user data
directory. If this argument is not given, the user data directory is simply a
directory within users
. The command will fail if this would cause the user
data to be stored inside /etc
or /usr/local/etc
; in this case, you need to
explicitly give a path for the user data.
By default, a password for the user is randomly generated. You can pass
--prompt-password
to input your own.
Renaming, aliasing, and deleting users
Crymap does not currently have special commands for these operations. Once created, users can be treated as regular file system objects. In particular:
-
A user can be renamed by simply renaming the entry under
users
. -
User aliases can be created by symlinking the additional alias to the user data directory. These symlinks are understood to allow the user to use all of them equivalently. For example, if
bob
is a symlink torobert
, the user can log in asrobert
and then send mail asbob
. -
Users can be deleted by removing their entry from
users
. -
Users can be disabled by renaming them to an illegal user name. The simplest way is to just prefix their name with
%
. You do need to deal with any aliases as well, though. -
A user can be imported from another Crymap installation by simply moving the user data directory (or a symlink thereto) into
users
.
Password Resets
When a user changes their password, a backup of the user configuration is
created in the tmp
directory within the user data directory, with a name of
the format config-backup-DATETIME.toml
. If necessary (for example, because
the user mistyped their new password), the password change can be undone by
replacing the user.toml
file at the top of the user data directory with the
backup file. Note that these backup files are automatically deleted after a
successful login 24 hours after the change was made.
If a user forgets their password, there is no recourse. Their data is gone forever. The best thing to do is to move their user data directory to somewhere else in case they remember the password later and create a new account for them.
Backups and Recovery
Backups
Starting with Crymap 2.0.0, each message and private key is stored in its own file, with all other data tracked in a couple SQLite databases.
Backup tooling does not need to support any exotic filesystem features, and even Windows filesystems will be able to carry the backup in tact.
Restoring from Backup
In most cases, simply restoring from backup should do the trick.
There are a few cases where a backup could be torn across an update:
-
Messages on the filesystem but not in the database. At least once a day, Crymap automatically checks for such messages. They will automatically appear in the user’s inbox.
-
Messages in the database but not in the filesystem. These will appear to the user as stub messages indicating the problem. The user must delete the entries themselves.
-
Corrupt SQLite database. You can remove
delivery.sqlite*
andmeta.sqlite.xex*
(important: ensure you remove the-journal
file too!), then copy one of themeta.sqlite.xex.*
backups from the user’sbackup
directory tometa.sqlite.xex
in the user’s directory. In the worst case, if you can’t get anything to work, you can removemeta.sqlite.xex*
entirely, which will result in all the user’s messages being in unread, inINBOX
, and in no particular order, but at least their mail will be there.
Objects from a user account are readable from only that user account. For example, in case of a system failure, you cannot set up a new system, create user accounts in it, and then expect to be able to drop data from the backups into those new user accounts. The user accounts themselves must be restored from backup.
Understanding the Filesystem
This section only covers what is useful to know when inspecting a Crymap installation with regular shell tools. For the nitty-gritty details, you’ll need to look at the Crymap source code’s internal documentation.
Crymap root
There are three files of significance at the Crymap root:
-
crymap.toml
. This is the “system configuration” and provides all of Crymap’s configuration other than logging and that which is represented by the filesystem. -
logging.toml
. If present, replaces syslog-based logging with something else. -
users
. Either a directory or a symlink to a directory which contains one entry for each Crymap user.
Refer to the configuration reference for the two configuration
files, and user management for the users
directory.
User data directory
The user data directory contains the following files:
-
backups
. This contains routine database backups produced atomically by Crymap. -
crymap-v1-files
. This contains the file trees that were present when the user was migrated from Crymap 1.x to Crymap 2.x. It will not be present for users that were created on Crymap 2.x, nor will it be recreated if removed. It is safe to remove entirely once you are sure you don’t need to roll back to Crymap 1.x. -
delivery.sqlite
anddelivery.sqlite-journal
. This is a cleartext SQLite database used for message delivery. Message delivery occurs without the user being logged in, and information about how such messages are to be added to the user’s account are written here. When the user is logged in, the database is processed and the messages are deposited into their final positions. If this database is removed, and pending messages will eventually show up in the inbox once Crymap determines that they have been orphaned. -
garbage
. This is a holdover from Crymap 1.x, and was used internally for deleting directories. It is safe to fully delete all its contents at any time. Crymap itself aggressively cleans it. -
keys
. A directory containing the user’s RSA keys. It is a critical part the user data needed for accessing the user’s mail. -
maintenance-run
. This is a marker file used to track the last time that a full maintenance pass was run on the account. This can be deleted safely, which will cause the next login to trigger a full maintenance pass. -
messages
. This contains the user’s actual email, one file per message. -
meta.sqlite.xex
andmeta.sqlite.xex-journal
. This is an encrypted SQLite database containing all information about the user’s messages and mailboxes. Atomic backups ofmeta.sqlite.xex
are routinely created underbackups
. If you restoremeta.sqlite.xex
from backup, you MUST also removemeta.sqlite.xex-journal
at the same time. -
tmp
. Used for temporary files and temporary markers. Crymap will automatically clean stale files out of this directory. It is not too important to back up (though it is also the destination for config backups which allow undoing password changes) but should not be cleaned manually unless there is some clear need to do so. -
user.toml
. This contains the user’s preferences and the data needed to derive their master key from their password. This file can be overwritten with an earlier backup version (either a backup created undertmp
or by some external backup system) to revert a password change.
OpenSSL mirrors
If Crymap is set up in a way that causes it to chroot, it will maintain a copy
of the system OpenSSL certificate store within each chroot. This will usually
manifest as an etc
or usr
directory within the users
directory or within
each user directory. This is needed so that certificates of external servers
can be validated when using outbound SMTP, as they must access these files
after Crymap has moved into the chroot and rendered the normal location of
these files inaccessible.
If possible, Crymap will create hard links, but will copy the files if the destination directory is on a different file system.
Tuning
Crymap’s performance tuning options are very limited; in fact, Crymap itself does not provide any at all.
Thread Count
As of Crymap 2.0.0, Crymap is always entirely single-threaded.
Memory Allocator
Crymap uses the host system’s malloc()
, so exactly how it gets tuned depends
on your system. Often, you can refer to malloc(3)
for information on how to
configure the allocator.
Crymap is very conservative about allocating memory and ensures that memory not
needed is freed expediently. The memory allocator itself may not be. In
particular, glibc’s malloc()
(used on most Linux installations) and jemalloc
(used on FreeBSD) will switch to a strategy optimised for multi-core
performance when running on a multi-core system. This can cause Crymap to use
dramatically more memory than it actually needs, potentially by a factor of as
much as 10. If running Crymap on a multi-core system with memory constraints,
it is useful to disable the multi-core allocation strategies.
With glibc malloc()
, this can be done by setting the environment variable
M_ARENA_MAX
to 1
.
With jemalloc, a similar thing can be done by setting MALLOC_CONF
to
narenas:1
.
Migration from Crymap 1.x
System
To upgrade from Crymap 1.x to 2.x, no configuration changes or user action is required. The recommended upgrade procedure is as follows:
- Disable all ways for Crymap server processes to be created.
- Terminate any remaining Crymap server processes.
- Upgrade the Crymap binary to 2.x.
- Reenable Crymap.
User
Crymap 2.x uses an entirely different data model than 1.x. When a user next logs in, they will automatically be migrated from the 1.x data model to the 2.x data model. This can take a few seconds to a few minutes, depending on the size of the account and the speed of the server.
The data migration process preserves all messages, message flags, and the mailbox hierarchy. It does not preserve message IDs of any kind or synchronisation states. The user’s email client will thus effectively start from a blank slate once migration completes and will need to resynchronise/redownload everything it wants to keep local.
There is no way to migrate users in advance, as the migration process requires the user’s credentials to proceed.
A user whose account is still on the 1.x data model can still receive mail from Crymap 2.x, though this mail will be added to the account under the 2.x data model.
Rollback
If you decide you need to roll back, the recommended procedure is as follows:
- Disable all ways for Crymap server processes to be created.
- Terminate any remaining Crymap server processes.
- Manually roll back any user accounts that had been upgraded.
- Downgrade the Crymap binary to 1.x.
- Reenable Crymap.
A user account that was migrated from the 1.x data model to the 2.x data model
can be identified by the presence of a crymap-v1-files
directory under the
user directory. A user can be rolled back to the 1.x model by running the
following commands in the user directory:
mv crymap-v1-files/* .
rmdir crymap-v1-files
rm -rf messages delivery.sqlite* meta.sqlite.xex*
This will reset the account to the state it was in before the migration, except
for changes to the user.toml
file (which would include password changes). If
the user.toml
file now has an [smtp_out]
section, you will need to remove
that manually with a text editor, or restore the user.toml
file from a backup
in tmp
if there is one or from another backup you had made.
Finishing touches
Once you are sure you won’t need to roll back, you can entirely remove the
crymap-v1-files
within each user directory.
User Guide
Mail server settings
Assuming your administrator has not indicated otherwise:
- Protocol: IMAPS or IMAP (POP not supported)
- Host/domain: Provided by administrator
- Port: 993
- Connection security: “SSL/TLS”, “Secure connection”; not “STARTTLS”
- Password/Authentication: “Normal”, “plain”
Do not proceed if you receive certificate or security warnings.
Mail submission settings must be provided by your administrator.
Changing your password or settings
To change your Crymap password or Crymap settings, you currently need the
crymap
program. Your administrator should provide this; if not, refer to the
installation subsection for ways to get
crymap
. (This is the same program used to run Crymap on the mail server.)
Changing your password
To change your IMAP password, run the below command, where USER
is the
username you use to log in to IMAP, and HOST
is the host or domain you use to
connect to IMAP:
crymap remote chpw --user=USER --host=HOST
The password change takes effect immediately, but does not terminate existing IMAP sessions.
Note that your Crymap password is independent of your other email password(s), if there are any. You need to change both. (It is possible to use different passwords for the separate systems too, if you prefer, though not all mail clients can work with such a configuration.)
If you find you need to undo the password change, the administrator can help you with that.
Changing key rotation settings
By default, Crymap rotates your mail encryption keys once per month. Rotation is controlled by key name templates, which are filled in with the current time. Changing these templates to include more or less of the date will have the effect of changing the key rotation frequency.
Below are some examples of setting the key rotation.
# Rotate once per day
crymap remote config --external-key-pattern "external-%Y-m-%d" \
--internal-key-pattern "internal-%Y-%m-%d" --user=USER --host=HOST
# Rotate once per year
crymap remote config --external-key-pattern "external-%Y" \
--internal-key-pattern "internal-%Y" --user=USER --host=HOST
# Rotate once per week
crymap remote config --external-key-pattern "external-%Y-%W" \
--internal-key-pattern "internal-%Y-%W" --user=USER --host=HOST
Outbound mail configuration
This section only applies if your site uses Crymap for outbound mail.
Normally, your mail client needs to “send” a message twice: Once to request it to be sent to the recipients, and again to save it into your “Sent” folder. You can configure Crymap to implicitly save sent messages instead, which will make the send process faster and smoother:
# By default you have a "Sent" folder. If you renamed it to something else,
# you need to use that name here. If the folder does not exist, your messages
# will *NOT* be saved!
crymap remote config --smtp-out-save=Sent --user=USER --host=HOST
If you do this, you’ll also need to configure your email client(s) to not also save a copy to the Sent folder themselves.
Whenever Crymap sends email to another server, it generates a “receipt” which includes technical details of the mail transaction. By default, if the transaction succeeds, the receipt is discarded, and if it fails, it is delivered to you as a message in your inbox. Both of these are configurable. In the example below, we assume you’ve created “Success” and “Failure” sub-folders under your default “Sent” folder.
# Configuring this will cause you to also get messages about mail successfully
# sent. If the folder you give here doesn't exist, they'll go to your inbox
# instead. These messages will be delivered already marked as "read".
# To disable, rerun with --smtp-out-success=''
crymap remote config --smtp-out-success-receipts=Sent/Success --user=USER --host=HOST
# Configuring this will change where failure receipts are sent. If the folder
# you give here doesn't exist, they'll go to your inbox instead. You cannot
# disable delivery of these notifications, as they are the only way to find out
# that your mail cannot be sent.
crymap remote config --smtp-out-failure-receipts=Sent/Failure --user=USER --host=HOST
Viewing the current configuration
To view the current configuration, simply run
crymap remote config --user=USER --host=HOST
IMAP Characteristics
This section is targeted at client developers or those with an interest in the lower-level IMAP details.
Conformance
Crymap fully conforms to the IMAP4rev1 (RFC 3501) and IMAP4rev2 (RFC 9051) standards. It also conforms to a number of extensions detailed in later sections.
Crymap passes the full Dovecot imaptest compliance suite excepting tests for extensions that Crymap does not implement.
Crymap was developed with the following priorities:
- Don’t corrupt the user’s mail. (The “prime directive”.)
- Be secure.
- Conform to standards.
- Be reasonably performant and light-weight.
In spite of the earlier paragraphs, there are cases where (1) and (3) come into conflict. Usually this is a result of a specification assuming all email messages are well-formed or pre-dating another specification change that makes compliance impossible without violating the prime directive, but there are also some issues with self-contradiction. These will all be discussed in individual sections where relevant.
Notwithstanding the few exceptions discussed, Crymap’s IMAP implementation conforms to the following standards:
- RFC 2045 MIME Extensions: Format of Internet Message Bodies
- RFC 2046 MIME Extensions: Media types
- RFC 2047 MIME Extensions: Message Header Extensions for Non-ASCII Text
- RFC 2157 UTF-7
- RFC 2177 (IDLE)
- RFC 2183 The Content-Disposition Header Field
- RFC 2231 MIME Parameter Value and Encoded Word Extensions: Character Sets, Languages, and Continuations
- RFC 2342 (NAMESPACE)
- RFC 2557 (Content-Location header field)
- RFC 2595 (PLAIN authentication)
- RFC 2971 (ID)
- RFC 3066 Tags for the Identification of Languages
- RFC 3282 Content Language Headers
- RFC 3348 (CHILDREN)
- RFC 3501 (IMAP4rev1)
- RFC 3502 (MULTIAPPEND)
- RFC 3516 (BINARY)
- RFC 3691 (UNSELECT)
- RFC 4315 (UIDPLUS)
- RFC 4731 (ESEARCH)
- RFC 4959 (SASL-IR)
- RFC 4978 (COMPRESS=DEFLATE)
- RFC 5161 (ENABLE)
- RFC 5182 (SEARCHRES)
- RFC 5253 (LIST-EXTENDED)
- RFC 5322 (Internet Message Format)
- RFC 5530 IMAP Response Codes
- RFC 5819 (LIST-STATUS)
- RFC 5918 Unicode Format for Network Interchange
- RFC 6154 (CREATE-SPECIAL-USE and SPECIAL-USE)
- RFC 6532 Internationalized Email Headers
- RFC 6851 (MOVE)
- RFC 6855 (UTF8=ACCEPT)
- RFC 7162 (CONDSTORE and QRESYNC)
- RFC 7888 (LITERAL+)
- RFC 8438 (STATUS=SIZE)
- RFC 8457 IMAP “$Important” Keyword and “\Important” Special-Use Attribute
- RFC 8474 (OBJECTID)
- RFC 8514 (SAVEDATE) since Crymap 2.0.0.
- RFC 9051 (IMAP4rev2) since Crymap 1.0.1.
Unicode support
Crymap is inherently a Unicode-aware, UTF-8-based application. All data which is returned to the client in a structured way (i.e., not as a raw byte stream) is internally managed as UTF-8.
If non-ASCII strings would be sent to a client that has not enabled UTF-8 support, one of two approaches is taken right before the data is put onto the wire, depending on the field:
-
The whole field may be put into RFC 2047 “encoded word” form, using base64-encoded UTF-8. Note that this means that returned “encoded words” are rarely in exactly the same form as they are in the original message, though the content remains the same.
-
Non-ASCII characters may be replaced by ‘X’. This is only done for fields that do not allow the previous strategy.
RFC 2184 is not implemented because it is impossible to implement its IMAP4 requirement at the same time as conforming to IMAP4rev1. Specifically, RFC 2184 would have the server decode encoded non-ASCII text in certain header parameters, but IMAP4rev1 does not allow these fields to contain non-ASCII content and the fields also may not contain encoded words. Instead, Crymap assumes the client will be able to deal with the parameters itself. This is likely to be a safe assumption, since this has always been an optional feature of IMAP servers, so clients need to be prepared to do it themselves either way.
Search uses Unicode “simple” case folding.
Limits
The maximum message size for APPEND
is hard-wired to 64MB.
Any operation that involves parsing message content will not process more than
20 levels of multipart or message/rfc822
nesting, nor will it process more
than 1000 different body parts.
Message header lines in excess of 64kB are not processed.
Search operations will consider up to 128kB of text. Non-text body parts are ignored for search.
A mailbox can have up to 4’294’967’294 message IDs allocated. Message IDs are allocated sequentially.
Mailboxes
All mailboxes other than INBOX
may have children. Nesting depth is dependent
on the host operating system’s maximum path length.
Mailbox names may not contain %
, *
, \
, /
, control characters or one of
a few Unicode control characters, and may not begin with .
or #
.
Mailbox names other than INBOX
are case-sensitive. INBOX
is always
upper-case.
The path delimiter is /
. Making a mailbox path “absolute” by prefixing it
with a /
has no effect. Duplicate /
characters in a name are ignored.
Non-normalised MUTF7 is accepted in mailbox names. Mailbox names are always returned in normalised MUTF7. MUTF7 is not returned on output nor transformed on input when the client has enabled UTF-8 support.
When UTF-8 support is enabled, including by way of IMAP4rev2, no normalisation
of mailbox names occurs other than removal of extraneous path delimiters and
the special case of INBOX
being case-insensitive. IMAP4rev2 has a
recommendation to send unsolicited LIST
responses when a client uses a
denormalised mailbox name. Due to the extremely limited utility of this feature
in the context of Crymap’s implementation, Crymap does not implement this
recommendation.
The \Marked
and \Unmarked
attributes are not used.
If a mailbox is deleted or renamed while a session has it open, the session will be terminated as soon as this occurrence is discovered.
When an account is created, the following mailboxes are made, with the shown special-use attributes.
Mailbox | Special-Use |
---|---|
INBOX | |
Archive | \Archive |
Drafts | \Drafts |
Sent | \Sent |
Spam | \Junk |
Trash | \Trash |
Messages
Crymap tolerates and preserves messages with arbitrary binary content.
Clients in sessions that have not yet observed the expungement of a message can continue to access it for 24hr after the expungement.
If a message contains 8-bit content in the headers and the client requests the
raw headers, Crymap will return that 8-bit content verbatim even if the client
has not enabled UTF-8 support. If a message contains binary content and the
client requests that binary content, even without using the BINARY
extension,
Crymap will return the binary content as-is because the IMAP specification
gives no other way to return the data without corrupting it. Both of these
issues are described in more detail in the descriptions of the UTF8=ACCEPT
and BINARY
extensions.
If messages contain non-DOS line endings, the line endings are returned as-is, so as not to corrupt anything. Crymap understands both UNIX and DOS line endings in messages. This is expected to be a non-issue, given that GMail actually served messages exclusively with UNIX line endings for years before anyone noticed. (Note though that when acting as a UNIX MDA, Crymap does convert line endings as they come in. This paragraph applies to messages that come in through IMAP or LMTP.)
Unsolicited FETCH
responses always include UID
and FLAGS
. If CONDSTORE
has been enabled, they include MODSEQ
as well.
Flags
All “system flags” are supported and permanent. \Recent
is fully implemented
and fully atomic. Arbitrary keywords can be created without limit.
Authentication
LOGIN
and AUTHENTICATE PLAIN
are both supported and have the same effect.
AUTHENTICATE
does not allow the authorisation and authentication usernames to
differ.
Extensions
RFC 8457
Crymap supports the $Important
keyword and the \Important
special-use
attribute.
RFC 5530
Extended status codes are used everywhere they make sense.
APPENDLIMIT
Crymap uses a fixed APPENDLIMIT
value which is reported in the capabilities
list.
BINARY
Binary literals are understood, but are not handled any differently than non-binary literals. (That is, Crymap accepts and properly handles binary content in all contexts.)
If a client requests a BINARY
body section, binary literal syntax is used iff
the response payload contains a NUL byte.
The BINARY
fetch feature is treated as orthogonal to the rest of the fetch
options, and Crymap currently allows combinations not allowed by the standard.
The BINARY
extension presents an odd conundrum: An IMAP4rev1 server is
required to report the actual content transfer encoding of a binary part
(binary
), and is required to return body parts in their original transfer
encoding when queried without BINARY
, but at the same time is forbidden from
returning binary data. In these cases, Crymap returns the binary data anyway,
since doing so is a lesser evil than corrupting the data itself.
Crymap does not perform any encoding changes of messages appended using binary literals.
The standard insinuates that if the client requests content transfer decoding of a body part, and the server finds that the body part does not use DOS newlines, it should convert the newlines to DOS newlines. Crymap does not do this, since that constitutes data corruption.
CHILDREN
If a client makes a non-extended LIST
command, \HasChildren
and
\HasNoChildren
mailbox attributes are returned implicitly.
COMPRESS=DEFLATE
Deflate-based compression may be enabled at any time.
Crymap is not aware of whether TLS itself is using compression and so will not reject enabling compression even if TLS compression is active.
CONDSTORE
This extension is fully implemented.
Atomic updates are performed with respect to all flags on a message and not just the flags being modified, since this is both simpler and more useful. (And seems like what clients expect — there has been some talk of using conditional store to maintain tri-state flags like Junk/NonJunk/(nothing) properly.)
Modseqs are 63-bit integers as required by the later QRESYNC extension.
If multiple messages are changed at once, they all receive the same Modseq.
Expunging a message increments the highest Modseq value.
Crymap does not return HIGHESTMODSEQ
response codes until CONDSTORE
is
enabled.
CREATE-SPECIAL-USE
The following special-use attributes are allowed: \Archive
, \Drafts
,
\Flagged
, \Junk
, \Sent
, \Trash
, \Important
. \All
is not allowed
since Crymap does not support an “all mail” view.
At most one special-use can be given to a mailbox. Crymap does not take action on special-use attributes except to return them, and does not prevent creating multiple mailboxes with the same special use.
ENABLE
This extension is fully implemented.
ESEARCH
This extension is fully implemented.
Crymap does not attempt to optimise searches for MIN
or MAX
alone.
ID
This extension is fully implemented.
The client-submitted identifying information is written into the server logs.
The [identification]
section of the server configuration can provide
additional attributes to be returned here. By default, Crymap returns name
and version
.
IDLE
This extension is fully implemented.
Notifications about changes are typically delivered within a millisecond of when they occurred (modulo client-server latency of course). Message creation and expungement and flag operations are all monitored.
Crymap does not strictly validate that the idle is terminated with “DONE”.
LIST-EXTENDED
This extension is fully implemented.
LIST-STATUS
This extension is fully implemented, but only because IMAP4rev2 requires it. No
optimisations are made over simply making separate STATUS
calls.
LITERAL+
This extension is fully implemented.
MOVE
This extension is fully implemented.
Moves occur as three separate atomic steps:
- Messages are added to the destination.
- Flags are set on the destination messages.
- Messages are removed from the source.
MULTIAPPEND
This extension is fully implemented.
Setting flags on the messages occurs in a separate step message insertion. (This is allowed by the spec since even setting the flags at all is only a SHOULD.)
NAMESPACE
Crymap does not have namespaces. The extension is implemented in that it returns a canned “no namespaces” response.
OBJECTID
This extension is fully implemented except for the optional THREADID
attribute, since Crymap does not support message threads.
Mailbox IDs always begin with M
, except for INBOX
s mailbox ID, which always
begins with I
. Email IDs always begin with E
.
All mailboxes support these attributes.
QRESYNC
This extension is fully implemented.
Crymap remembers all expunge events.
SASL-IR
This extension is fully implemented.
SAVEDATE
This extension is fully implemented as of Crymap 2.0.0.
When migrating from the Crymap 1.x message store, the SAVEDATE
of each
message is initialised to the modified time of the file storing the message.
SEARCHRES
This extension is fully implemented.
SPECIAL-USE
This extension is fully implemented.
See CREATE-SPECIAL-USE for a reference on what attributes are supported.
STATUS=SIZE
This extension is fully implemented.
In versions of Crymap prior to 2.0.0, this was only implemented to the letter of the standard and not the spirit due to efficiency concerns: Each message was assumed to be 4GB in size and this size was simply multiplied by the message count. Crymap 2.0.0 is able to track message sizes efficiently, so this limitation no longer applies.
UIDPLUS
This extension is fully supported.
No mailboxes have the UIDNOTSTICKY
attribute.
UNSELECT
This extension is fully implemented.
UTF8=ACCEPT
The useful part of this extension is implemented. Clients using the extension cannot observe aspects (arguably) unimplemented. To clients not using the extension, Crymap is the same as any IMAP implementation that does not support this extension.
Once a client enables this extension, mailbox names are returned and accepted in UTF-8, and MUTF7 interpretation no longer occurs. Envelope and body structure data is returned in UTF-8, with all encoded words decoded.
The “UTF-8 literals” added by this extension are handled no differently than normal literals. The standard suggests that the server should do something to “downgrade” UTF-8 messages for the sake of clients not using this extension. Crymap does not do this, since it would violate the prime directive, and is also not useful since nearly all clients are already prepared to encounter 8-bit characters where they are not formally allowed by the older 7-bit standard.
The standard requires that non-UTF8 literals containing 8-bit characters in the message headers are rejected. Crymap also does not do this, since it is actively harmful, leaving users without the ability to copy their existing messages into Crymap, and as mentioned in the previous paragraph, clients are able to deal with such messages anyway.
XCRY
Crymap-specific extension. This entails several commands:
XCRY PURGE
No arguments.
All expunged messages in the selected mailbox are removed from the host filesystem immediately, making them inaccessible to concurrent sessions that have not yet observed the expungement.
This is mainly used as part of Crymap’s internal test suite.
XCRY GET-USER-CONFIG
No arguments.
Retrieves the current user configuration.
Response:
* XCRY USER-CONFIG (capabilities...)
internal-key-pattern
external-key-pattern
password-changed-datetime
key value key value [...]
The GET-USER-CONFIG
subcommand was poorly designed in Crymap 1.x.
The Crymap 1.x capabilities are INTERNAL-KEY-PATTERN
, EXTERNAL-KEY-PATTERN
,
and PASSWORD
, which correspond to the ability to pass each of those settings
to SET-USER-CONFIG
. Those three settings are guaranteed to be present, in
that order, immediately after the capabilities in the USER-CONFIG
response.
Each is an nstring
.
Beyond the Crymap 1.x settings are other settings. Each key
is an atom
naming the setting, and the value
is an nstring
giving the current value of
that setting. The presence of a setting in this list implies that it can be
passed to XRY SET-USER-CONFIG
.
Crymap 2.0.0 adds the SMTP-OUT
capability. This indicates the presence of the
XCRY SMTP-OUT
subcommand and the following new settings:
SMTP-OUT-SAVE
, mailbox into which sent messages are implicitly saved (defaultNIL
, meaning no implicit saving happens)SMTP-OUT-SUCCESS-RECEIPTS
, mailbox into which receipts for successfully-delivered messages are saved (defaultNIL
, meaning no delivery of success receipts)SMTP-OUT-FAILURE-RECEIPTS
, mailbox into which receipts for unsuccessfully-delivered messages are saved (defaultNIL
, which is the same as"INBOX"
)
capabilities
provides a list of valid tokens that can be passed to XCRY SET-USER-CONFIG
.
XCRY SET-USER-CONFIG
Arguments: key value [key value [...]]
Updates the given key-value pairs in the configuration. Each key
is an
atom
; each value
is an nstring
. Using this to “configure” PASSWORD
is
also how password changes are done.
Response:
* XCRY BACKUP-FILE "filename"
The returned filename should be shown to the user to let them know what file to use to undo the change.
XCRY SMTP-OUT FOREIGN-TLS LIST
No arguments.
Produces a list of the TLS requirements imposed on outbound SMTP domains.
Example responses:
* XCRY SMTP-OUT FOREIGN-TLS secure.example.com STARTTLS VALID-CERTIFICATE "TLS 1.3"
* XCRY SMTP-OUT FOREIGN-TLS insecure.example.com NIL
XCRY SMTP-OUT FOREIGN-TLS DELETE
Arguments: domain [domain ...]
Each domain
is an astring
naming a domain whose TLS requirements are to be
forgotten. The next attempt to send mail to that domain will not require any
particular TLS features to be present.
XCRY SMTP-OUT SPOOL EXECUTE
Arguments: message-id
message-id
is an astring
naming a spooled message ID which should be
retried.
Currently, spooled message IDs can only be found by the user in message failure receipts.
XLIST
Implements the XLIST
command, which was developed for GMail before
LIST-EXTENDED
was standardised. Some clients may still use this instead of
the extended LIST
command.
XVANQUISH
Crymap-specific extension.
Adds the XVANQUISH
command. This command takes one argument, a UID sequence
set. The given messages are immediately expunged without having to go through
the \Deleted
dance first.
This is not under the XCRY
umbrella since it could be useful to others.
XYZZY
This is obviously a very important extension for GMail compatibility.
Miscellaneous
Crymap always returns CAPABILITY
response codes to the OK
used as a
greeting upon connection and after successful authentication.
Crymap accepts UNIX line endings in IMAP, but always outputs DOS line endings.
Extensions not implemented
Extensions in this list were deliberately excluded because they were not useful or harmful.
ACL, LIST-MYRIGHTS, RIGHTS=
Since users are strictly bound to their own mailboxes, permissions don’t make much sense for Crymap.
CATENATE, URL-PARTIAL, URLAUTH, URLAUTH=BINARY
Not implemented due to higher complexity with low benefit.
URLAUTH is feasible to implement, as it would work by encoding the session key of the message in question in the auth string. This would probably be sufficient to make it work with BURL. However, it’s not a feature that the author would get use of any time soon.
CONVERT
Insanity.
It also looks like, quite possibly, literally nobody has ever implemented this for server or client.
I18NLEVEL=
Requires use of an out-dated, non-standard algorithm for Unicode collation and folding.
SORT, ESORT, THREAD
These concerns are much better handled by the client. Now that QRESYNC exists, clients can cheaply keep envelope data synchronised and not only do these operations themselves, but do them using up-to-date, standardised collation algorithms which take into account the user’s locale.
Secondarily, these add a decent amount of memory overhead and aren’t something the author would ever get use of.
UNAUTHENTICATE
Incompatible with the way Crymap chroots into the user data directory in traditional UNIX-style deployments.
WITHIN
Should Crymap ever implement the CONTEXT extensions, this extension has a pathological interaction with them, and itself offers very little benefit.
QUOTA
Besides being useless to the author, it’s also unclear how this should really work, since mail delivery does not have access to the information it would need to maintain the quota.
SMTP/LMTP Characteristics
Conformance
Crymap’s SMTP/LMTP implementation is believed to conform to:
- RFC 1652 (8BITMIME)
- RFC 1830 (BINARY)
- RFC 1854 (PIPELINING)
- RFC 1870 (SIZE)
- RFC 1893 and RFC 2034 (ENHANCEDSTATUSCODES)
- RFC 2033 (LMTP)
- RFC 3207 (STARTTLS)
- RFC 4954 (AUTH PLAIN)
- RFC 5321 (SMTP)
- RFC 6531 (SMTPUTF8)
Inbound variants
SMTP delivery
STARTTLS is fully supported, but not required.
Crymap will always evaluate DMARC/DKIM/SPF, but will not reject messages on that basis unless enabled in the configuration. If enabled, it rejects it in the SMTP transaction.
Crymap does not generate any DMARC reports.
Attempts to authenticate on the inbound SMTP port will always be rejected.
Message delivery will always be rejected for any recipient domain which is not explicitly defined in the server configuration, even if the server is otherwise configured to not consider the domain during delivery, so that the Crymap server does not appear to be an open relay.
SMTP submission
STARTTLS is fully supported for connections that start in cleartext and is required before any attempt to authenticate will be allowed.
The BINARYMIME
capability is not reported to the client in SMTP submission as
Crymap is not able to downgrade binary messages for transmission to servers
that do not support them.
Messages cannot be submitted until the user has successfully authenticated.
Crymap validates that the return path and the From
header all reference the
authenticated user and correspond to a domain which is explicitly defined in
the server configuration.
LMTP
STARTTLS is fully supported, but not required.
DMARC/DKIM/SPF are not evaluated; that should be handled by whatever is fronting LMTP. If Crymap is configured to ignore the email domain when identifying users, delivery will be accepted for any domain.
Attempts to authenticate over LMTP will always be rejected.
General notes
DOS-style line endings are not required.
The DATA
command will perform line-ending conversion from UNIX to DOS
newlines in two conditions:
- The client is using UNIX newlines at the SMTP level.
- The first line ending in the data is a UNIX newline.
If neither of these conditions are met, binary data can be correctly sent even
through a plain DATA
command, as long as the other end uses the same very
strict interpretation of dot-stuffing.
Line ending conversion is never performed on messages sent through the
CHUNKING
extension.
Maximum line length is 65534 bytes.
For SMTP delivery and SMTP submission, which must inspect the header block, the message header block may not exceed 256kB or the message will be rejected. LMTP does not inspect the header block and so has no such limit.
VRFY
and EXPN
are not supported and return constant hardwired responses.
The BODY=
argument to MAIL FROM
is entirely ignored. For SMTP submission,
Crymap identifies the appropriate transfer type by inspecting the message
itself.
The same limit used for IMAP APPEND is enforced for SMTP and LMTP.
Outbound SMTP
The outbound SMTP implementation is simplistic and only suitable for low-volume sites.
A given SMTP session will only be used to send one message (but possibly to multiple recipients).
Crymap remembers the best TLS characteristics it has seen for an email domain (not a particular SMTP server), and will abort the transaction if the connection it obtains cannot meet those same characteristics:
- Whether or not the server supports TLS at all;
- Whether the server provides a valid certificate;
- The TLS version.
Crymap will include the exact size of the message in the MAIL FROM
command if
the server supports the SIZE
extension. If the server supports the SIZE
extension and indicates a definite size limit which is smaller than the size of
the message, Crymap will fail the transaction without attempting it.
Crymap is not capabable of performing any kind of “downgrading” of a message before sending it. In general, the assertion is that servers which do not support transport of 8-bit content simply do not exist.
-
If a binary message is to be sent (which can only occur if the user agent which submitted the message ignored the lack of the
BINARYMIME
capability), Crymap will reportBODY=BINARYMIME
if the remote server advertisesBINARYMIME
,BODY=8BITMIME
if the remote server does not advertiseBINARYMIME
but does advertise8BITMIME
, and noBODY=
argument if the server supports neither extension. -
If an 8-bit message is to be sent, Crymap will report
BODY=8BITMIME
if the remote server advertises8BITMIME
, and noBODY=
argument otherwise.
The presence of the SMTPUTF8
capability has no effect. For a message that
nominally requires that capability, Crymap will attempt to send it anyway and
allow the server to decide whether or not it understands the SMTP commands and
the message itself.
Crymap will always transfer the message with BDAT
if the server supports the
CHUNKING
extension.
Crymap never takes advantage of pipelining or enhanced status codes.