Map functions
A Map is a set of key-value pairs and the WEL type for a JSON object. Map functions read its keys and values, combine one map with another, remove fields, and convert between a map and a list so that list functions can operate on it.
Keys are String by default. Insertion order is preserved, so a map built in a deliberate order serializes in that order.
_ is the input your expression receives in these examples. Refer to Input variables for more information.
Call a map function
Call a function directly, or pipe a value into it with >>:
keys(_.customer)
_.customer >> keysRead a field with a period (.), or with brackets ([]) when the key is computed or isn't a bare identifier:
_.customer.email
_.customer['email']Reading a key that doesn't exist raises E104. Use has_key? to test first, or the try-fallback operator |? to supply a default:
_.customer.nickname |? 'none'Read keys and values
The following functions read a map's keys and values without changing it:
keys
Returns the keys as a list, in insertion order.
keys(map)| Parameter | Description |
|---|---|
| map | The map to read. |
Get the keys of a map
The following example returns the keys of a map, in insertion order:
Formula
keys({sku: 'WID-1', qty: 3})Output
["sku", "qty"]values
Returns the values as a list, in insertion order. Values that are Skip are omitted.
values(map)| Parameter | Description |
|---|---|
| map | The map to read. |
Get the values of a map
The following example returns the values of a map, in insertion order:
Formula
values({sku: 'WID-1', qty: 3})Output
["WID-1", 3]has_key?
Returns true if the key is present.
A test for the key isn't the same as a test for a value. A key present with a null value returns true here. Use present? instead to test for a usable value.
has_key?(map, key)| Parameter | Description |
|---|---|
| map | The map to test. |
| key | The key name, as a String. Non-string data types raise E100. |
A present key returns true
The following example tests a key that is present in the map:
Formula
has_key?({sku: 'WID-1'}, 'sku')Output
trueA missing key returns false
The following example tests a key that isn't present in the map:
Formula
has_key?({sku: 'WID-1'}, 'qty')Output
falseat
Returns the value at an exact key. A key present with a null value returns null rather than raising an error. at also works on a List, where it takes an Integer index instead of a key.
at(map, key)| Parameter | Description |
|---|---|
| map | The map to read from. |
| key | The key name as a String. A missing key raises catchable E104. Non-string data types raise E100. |
Get the value at a key
The following example gets the value at a key:
Formula
at({sku: 'WID-1', qty: 3}, 'qty')Output
3Choose fields
The following functions narrow a map to a smaller, safer set of fields:
pluck
Returns a map containing only the named keys, in the order keys lists them rather than the map's own order. pluck skips keys that aren't present and deduplicates keys named more than once.
Use it to narrow a large inbound record to the fields a destination actually accepts.
pluck(map, keys)| Parameter | Description |
|---|---|
| map | The map to read from. |
| keys | A list of key names to keep. Non-string data types raise E100. |
Keep only the named keys
The following example keeps only the named keys from the map:
Formula
pluck({id: 1, secret: 'x', name: 'a'}, ['id', 'name'])Output
{id: 1, name: "a"}A missing key is skipped rather than raising an error
The following example plucks a key that isn't present, which is skipped rather than raising an error:
Formula
pluck({id: 1}, ['id', 'missing'])Output
{id: 1}except
Returns a map with the named keys removed. except skips keys named in keys but not present in the map.
pluck and except are opposites, and the choice between them matters for safety. pluck is an allowlist, so a new field appearing upstream is dropped by default. except is a denylist, so a new field is forwarded by default. Prefer pluck when leaking a newly added field is the concern.
except(map, keys)| Parameter | Description |
|---|---|
| map | The map to read from. |
| keys | A list of key names to remove. Non-string data types raise E100. |
Remove the named keys
The following example removes the named keys from the map:
Formula
except({id: 1, secret: 'x', name: 'a'}, ['secret'])Output
{id: 1, name: "a"}Use case: Drop sensitive fields before forwarding
An inbound user record carries credentials that shouldn't reach the destination system. Use except to drop the sensitive fields before forwarding the record:
Input
{
"user": {
"id": "U-1",
"email": "nur@example.com",
"password": "hunter2",
"ssn": "111-22-3333"
}
}Formula
_.user >> except(['password', 'ssn'])Output
{"id": "U-1", "email": "nur@example.com"}This use of except is the denylist form, and it's only safe while you know every field the source can send. pluck(['id', 'email']) is the safer expression of the same intent if the source may add fields.
Combine maps
The following functions combine two maps into one. merge combines only the top-level keys: nested maps replace each other completely. deep_merge and deep_merge_by merge nested maps at any depth.
merge
Combines two maps. The second map wins when both define the same key, even when the winning value is null. merge places a key defined in both maps in the position it had in the first map, and appends a key only the second map at the end.
merge(first, second)| Parameter | Description |
|---|---|
| first | The base map. |
| second | The map whose values take precedence. |
Merge two maps, second wins on conflict
The following example merges two maps, where the second map's value wins on a shared key:
Formula
merge({sku: 'WID-1', qty: 1}, {qty: 5})Output
{sku: "WID-1", qty: 5}Use case: Apply overrides to a set of defaults
A set of defaults needs caller-supplied overrides applied on top, so anything supplied wins and anything omitted falls back. Use merge with the defaults first and the caller's values second:
Input
{
"overrides": {"currency": "EUR", "notify": true}
}Formula
merge(
{currency: 'USD', locale: 'en-US', notify: false},
_.overrides
)Output
{"currency": "EUR", "locale": "en-US", "notify": true}merge combines only the top-level keys. A nested map in the second argument replaces its counterpart outright rather than being merged into it. Use deep_merge or deep_merge_by to merge nested maps.
deep_merge
Combines two maps. Map keys merge values instead of replacing each other.
deep_merge(first, second, options)| Parameter | Description |
|---|---|
| first | The base map. A non-map argument raises E100. |
| second | The map whose values take precedence, even when the winning value is null. A non-map argument raises E100. |
| options | An optional map to control the merge.
options raises E100. An unrecognized key raises E225. |
Merge a shared nested map
The following example merges two maps that share a nested map:
Formula
deep_merge({retry: {attempts: 3, delay: 100}}, {retry: {attempts: 5}})Output
{retry: {attempts: 5, delay: 100}}Concatenate lists instead of replacing them
The following example appends the list from second to the list from first instead of replacing it:
Formula
deep_merge({a: [1, 2]}, {a: [3]}, {lists: 'concat'})Output
{a: [1, 2, 3]}deep_merge_by
Combines two maps. Map keys merge values instead of replacing each other. deep_merge_by calls a resolver for non-map conflicts. A key present in only one map passes through unchanged, without calling resolver.
deep_merge_by(first, second, resolver, options)| Parameter | Description |
|---|---|
| first | The base map. A non-map argument raises E100. |
| second | The map to combine with first. A non-map argument raises E100. |
| resolver | A lambda that decides the value to keep for a conflict deep_merge_by doesn't merge automatically. resolver receives three parameters.
Skip to omit the key entirely. resolver raises E105 if you don't provide exactly three parameters, and E100 if it returns a function instead of a value. |
| options | An optional map to control the merge.
options raises E100. An unrecognized key raises E225. |
Combine conflicting values instead of replacing them
The following example adds conflicting counts together instead of keeping only the value from second:
Formula
deep_merge_by({counts: {ok: 2}}, {counts: {ok: 3}}, (path, first, second) ~> first + second)Output
{counts: {ok: 5}}Drop a conflicting field instead of resolving it
The following example uses path to drop one conflicting field and keep the value from second for the rest:
Formula
deep_merge_by({a: 1, b: 2}, {a: 9, b: 3}, (path, first, second) ~> if path == ['a'] then Skip else second)Output
{b: 3}Convert between maps and lists
These four functions are how a map reaches the list functions. entries and pairs go one way. from_entries and from_pairs come back.
entries
Converts a map to a list of {key, value} maps.
entries(map)| Parameter | Description |
|---|---|
| map | The map to convert. |
Convert a map to a list of key/value maps
The following example converts a map to a list of {key, value} maps:
Formula
entries({sku: 'WID-1', qty: 3})Output
[{key: "sku", value: "WID-1"}, {key: "qty", value: 3}]from_entries
Converts a list of {key, value} maps back to a map. The inverse of entries.
from_entries(list)| Parameter | Description |
|---|---|
| list | A list of maps, each with a key and a value. An element missing a key or value field raises E100. A non-String key, such as an Integer or a Boolean, is coerced to a String. Two elements that coerce to the same key raise catchable E223. |
Convert a list of key/value maps back to a map
The following example converts a list of {key, value} maps back to a map:
Formula
from_entries([{key: 'sku', value: 'WID-1'}])Output
{sku: "WID-1"}pairs
Converts a map to a list of two-element [key, value] lists. Pairs whose value is Skip are omitted.
pairs(map)| Parameter | Description |
|---|---|
| map | The map to convert. |
Convert a map to a list of key/value pairs
The following example converts a map to a list of two-element [key, value] lists:
Formula
pairs({sku: 'WID-1', qty: 3})Output
[["sku", "WID-1"], ["qty", 3]]from_pairs
Converts a list of two-element [key, value] lists back to a map. The inverse of pairs.
from_pairs(list)| Parameter | Description |
|---|---|
| list | A list of two-element lists. A malformed element raises E100. A non-String key is coerced to a String. Two elements that coerce to the same key raise catchable E223. |
Convert a list of key/value pairs back to a map
The following example converts a list of two-element [key, value] lists back to a map:
Formula
from_pairs([['sku', 'WID-1']])Output
{sku: "WID-1"}FILTER A MAP WITHOUT CONVERTING IT
filter_by accepts a map directly. The lambda receives two parameters, the key and the value, and the result is a map:
{a: 1, b: 2} >> filter_by((k, v) ~> v > 1)Flatten nested maps
The following function collapses a nested map into a single level, joining the path into one key:
flatten_map
Collapses nested maps into a single level, joining the path into one key.
The walk is depth-first in insertion order. Only maps are descended into. A list is treated as a leaf value and kept whole. An empty nested map is also a leaf.
Two nested keys can collapse onto the same flattened key. This raises catchable E223 rather than silently keeping one write over the other.
flatten_map(map, options)| Parameter | Description |
|---|---|
| map | The map to flatten. |
| options | Optional map to control the flattening. An unrecognized option key raises E225 rather than being ignored.
|
Flatten a nested map with the default separator
The following example flattens a nested map, joining the path with the default . separator:
Formula
flatten_map({customer: {name: 'Nur', address: {city: 'London'}}})Output
{"customer.name": "Nur", "customer.address.city": "London"}Flatten a nested map with a custom separator
The following example flattens a nested map using a custom separator:
Formula
flatten_map({a: {b: 1}}, {sep: '_'})Output
{a_b: 1}Use case: Flatten a record for a CSV row
A CSV destination needs one column per field, but the source is nested two levels deep. Use map functions to flatten the record into a single level:
Input
{
"record": {
"id": "C-1",
"customer": {
"name": "Nur",
"address": {"city": "London", "postcode": "E1"}
}
}
}Formula
_.record >> flatten_mapOutput
{
"id": "C-1",
"customer.name": "Nur",
"customer.address.city": "London",
"customer.address.postcode": "E1"
}The flattened keys are exactly the column headers the destination expects. Pass {sep: '_'} if the target doesn't accept dots in header names.
Related
- List functions: What to do with
entries,pairs,keys, andvalues. - Deep map functions: Reach through nesting rather than flattening it.
- Data types:
Map,Skip, andnull. - Error codes: Troubleshoot job failures such as
E104,E105,E218, andE223.
Last updated: