Encoding functions ​

Encoding functions convert between text and bytes using a named character encoding, and handle the encoded-word form used in email headers.

These functions matter when a system doesn't speak UTF-8, such as a mainframe export in Shift_JIS, a legacy CSV in Windows-1252, or an email header carrying an accented name. A wrong encoding produces the garbled text that shows up as é where an é should be.

ENCODING ISN'T ESCAPING

Encoding is about which bytes represent a character. Escaping is about which characters are safe inside a format. An accented name sent to a CSV needs both, and neither substitutes for the other.

Text to bytes and back ​

The following functions convert between text and bytes with a specified character encoding:

encode_string ​

Converts a string to Binary using a named encoding.

Use encode_string instead of Binary() when text can contain non-ASCII characters. Binary() raises E008 for non-ASCII input instead of selecting an encoding.

wel
encode_string(text, encoding)
ParameterDescription
textThe string to encode.
encodingAn encoding name, such as UTF-8 or Shift_JIS. A name suffixed with -BOM, such as UTF-8-BOM, prepends a byte-order mark. A name suffixed with -NFC, -NFD, -NFKC, or -NFKD, such as UTF-8-NFC, normalizes the text to that form before encoding. An unrecognized name raises E206. A character the target encoding can't represent raises E227.
Encode an accented string as UTF-8

The following example encodes an accented string as UTF-8 bytes:

Formula

wel
encode_string('café', 'UTF-8')

Output

wel
0x"636166C3A9"

The é occupies two bytes, C3A9, so the 4-code-point string café encodes to 5 bytes. length counts code points, grapheme_length counts user-perceived characters, and byte_length counts bytes. The three can differ for anything outside ASCII.

decode_string ​

Converts Binary to a string using a named encoding. This conversion strips leading byte-order marks.

wel
decode_string(binary, encoding, options)
ParameterDescription
binaryThe bytes to decode. Data types other than Binary raise E100.
encodingThe encoding the bytes are in. An unrecognized name raises E206.
optionsOptional map to control how invalid sequences are handled. invalid: 'replace' substitutes a replacement character, or 'skip' drops them. Other invalid values raise E225. An unrecognized key in options raises E225.

An invalid sequence raises an error by default. Silently replacing bytes instead would turn a data problem into a corrupted record that nobody notices.

Decode UTF-8 bytes back to a string

The following example decodes UTF-8 bytes back to the original string:

Formula

wel
decode_string(encode_string('café', 'UTF-8'), 'UTF-8')

Output

text
café

decode_string_by_content_type ​

Decodes bytes using the charset named in an HTTP Content-Type header, rather than one you supply.

Use it when handling an HTTP response whose encoding you don't control.

wel
decode_string_by_content_type(binary, content_type, options)
ParameterDescription
binaryThe bytes to decode. Data types other than Binary raise E100.
content_typeThe Content-Type header value.
optionsOptional map specifying a fallback encoding to use when the header names none. An unrecognized key in options raises E225.

Check before converting ​

The following functions test whether text or bytes support a specified encoding:

ascii_string? ​

Returns true if the string contains only ASCII characters.

wel
ascii_string?(text)
ParameterDescription
textThe string to test.
A plain string is ASCII

The following example tests a plain ASCII string:

Formula

wel
ascii_string?('plain')

Output

wel
true
An accented string isn't ASCII

The following example tests a string containing an accented character:

Formula

wel
ascii_string?('café')

Output

wel
false

convertible_to_encoding? ​

Returns true if the given encoding can represent the string without losing anything.

Check convertibility before sending text to a system with a restricted character set. It's better to reject a record here than to deliver one with characters silently replaced.

wel
convertible_to_encoding?(text, encoding)
ParameterDescription
textThe string to test.
encodingThe target encoding. An unrecognized encoding raises E206.
Test whether an accented string fits ASCII

The following example tests whether an accented string can convert to ASCII without loss:

Formula

wel
convertible_to_encoding?('café', 'ASCII')

Output

wel
false

valid_string? ​

Returns true if the bytes are valid in the given encoding and can be decoded.

wel
valid_string?(binary, encoding)
ParameterDescription
binaryThe bytes to test. Data types other than Binary raise E100.
encodingThe encoding to test against. An unrecognized encoding raises E206.
Valid UTF-8 bytes

The following example tests bytes that are valid UTF-8:

Formula

wel
valid_string?(Binary('AB'), 'UTF-8')

Output

wel
true

encodings ​

Lists the encoding types that other encoding functions accept. Encoding names are case-insensitive.

wel
encodings(include_synonyms)
ParameterDescription
include_synonymsOptional boolean flag. Controls whether the response contains accepted synonyms in addition to the canonical names of each encoding type, such as utf8 as a synonym to UTF-8. Defaults to false. Non-boolean data types raise E100.

Email headers ​

An email header can only carry ASCII. RFC 2047 wraps anything outside that range in encoded-words that look like =?UTF-8?B?...?=.

encode_mime_header ​

Encodes text as RFC 2047 encoded-words for an email header.

wel
encode_mime_header(text, encoding, mode)
ParameterDescription
textThe header text to encode.
encodingThe character encoding to use. An unrecognized encoding raises E206. A character the encoding can't represent raises E227.
modeOptional encoding mode. Defaults to 'B' (base64). An unrecognized mode raises E225.

The result splits the output into multiple encoded-words when one word would exceed the 75 character limit RFC 2047 sets for encoded words.

Encode an accented name for a header

The following example encodes an accented string as an RFC 2047 encoded-word:

Formula

wel
encode_mime_header('café', 'UTF-8')

Output

text
=?UTF-8?B?Y2Fmw6k=?=

decode_mime_header ​

Decodes RFC 2047 encoded-words in an email header back to text. decode_mime_header drops whitespace between adjacent encoded-words. Adjacent words can use different charsets.

wel
decode_mime_header(text)
ParameterDescription
textThe header value to decode. A malformed encoded word raises E226. An unrecognized charset raises E206.
Decode an RFC 2047 encoded-word

The following example decodes an RFC 2047 encoded-word back to text:

Formula

wel
decode_mime_header('=?UTF-8?B?SGVsbG8=?=')

Output

text
Hello

Use case: Prepare a name for a legacy ASCII-only system ​

The destination exclusively accepts ASCII. Use convertible_to_encoding? to check each name before sending, and route the ones that won't fit rather than delivering corrupted text:

Input

json
{
  "names": ["Kalani Park", "Noam Gupta", "José Álvarez"]
}

Formula

wel
let parts = _.names >> partition_by(n ~> convertible_to_encoding?(n, 'ASCII'))
do {
  send: parts >> at(0),
  needs_review: parts >> at(1)
}

Output

json
{
  "send": ["Kalani Park", "Noam Gupta"],
  "needs_review": ["José Álvarez"]
}

Check convertibility before transmission so a person can correct the record instead of receiving corrupted text such as Jos? lvarez.

Last updated: