Speck

A library for input validation and protocol documentation with a focus on being lightweight and tightly focused, designed for embedded systems and web applications.

Installation

The package can be installed by adding speck to your list of dependencies in mix.exs:

def deps do
  [
    {:speck, "~> 1.1.0"}
  ]
end

Add the Speck compiler to your mix.exs project config:

compilers: [:speck] ++ Mix.compilers

Configuration

Create schemas in the protocol directory at the root of your Elixir project. Schemas are compiled, and therefore should use the .ex extension. Schemas can be grouped into subfolders if your project interfaces with more than one transport. Avoid putting your schemas in lib, as they use a different compiler than standard Elixir files. The schema directory can be changed in your project's mix config:

config :speck, schema_path: "my_schemas"

Philosophy

Speck is designed to be the validation layer for an application. The key here is "layer": Your application should employ a layered architecture to use Speck effectively. Speck should sit between the external input and business logic layers. In an MVC framework this would be between the controller and model. The layers should have hard boundaries, meaning function calls can only happen sideways or to the layer directly below. Input and return values should pass through a transformation when crossing layer boundaries, preventing a layer from leaking through its neighbors.

Phoenix / Ecto users: Although Speck schemas may feel familiar compared to Ecto schemas, it is important to understand that the Speck design pattern is completely different. Phoenix uses a leaky abstraction to pass input params into a changeset / schema, which is used all the way down to the database. With Speck, the schema should represent the shape of the input, NOT the shape of the database. Validation should happen in the controller or live view, with the attributes of a valid schema then being passed to the context layer (business logic). The context layer should be responsible for constructing the application's internal representation of the data, which could be a struct or Ecto schema.

The structs used for validation should be different from the ones passed around internally. In an embedded system the structs would represent messages from another system. In a web application the structs would represent form data from an HTML page.

A good way to determine which layer a struct is in is to determine if it is a noun (Device) or a verb (AddDevice). Verbs are effects on your system, which are messages with payloads to validate. Nouns are a representation of an object internal to your system, and are below the validation layer.

# Validation layer

%MQTT.AddDevice{}
%MQTT.RemoveDevice{}

# Model layer

%Device{}

The schemas can also be versioned.

# mqtt/v1/add_device.ex
%MQTT.V1.AddDevice{}

# mqtt/add_device.v1.ex
%MQTT.AddDevice.V1{}

Transform data when crossing layers. In the input handler, perform the message validation. If the payload is invalid, it can fail fast and return the error(s) upstream. If it succeeds, transform the message into internal data (struct) and pass it to the business logic layer.

def input_received(params) do
  case Speck.validate(MQTT.AddDevice.V1, params) do
    {:ok, message} ->
      device = Device.create!(message.id, message.rs485_address)
      {:ok, device}

    {:error, errors} ->
      {:error, errors}
  end
end

Schemas as documentation

Speck supports the "documentation as code" philosophy and is designed to document the input payloads with its description language. The same file that explains the input to an engineer is the same one executed to validate the input. This prevents documentation for people from going out of date with the source code. The schemas can also be used for collaboration when designing or explaining a protocol. They are designed to be reasonably comprehensible by non-Elixir engineers.

Schemas can't be nested. They are designed to be the complete representation of a message. If you find yourself trying to nest schemas, it is likely a code smell for something you should be doing in the business logic or storage layers.

struct MQTT.AddDevice.V1

name "add_device"

attribute :uuid,           :string,  format: ~r/\A\d{5}\-\d{5}\-\d{5}\-\d{5}\-\d{5}\z/
attribute :type,           :atom,    values: [:temperature, :humidity, :air_quality]
attribute :rs485_address,  :integer, min: 1, max: 255
attribute :serial_number,  :string,  length: 16
attribute :wifi_ssid,      :string,  optional: true
attribute :low_power_mode, :boolean, optional: true
attribute :dns_servers,    [:string]
attribute :user_data,      :any

attribute :metadata do
  attribute :location,        :string
  attribute :department,      :string
  attribute :commissioned_at, :datetime

  attribute :ports do
    attribute :rs485, :integer, strict: true
  end
end

attribute [:sensors], optional: true do
  attribute :type,    :atom
  attribute :address, :integer
end

Schema syntax

• struct - Name of the Elixir struct this schema will compile to.
• name (optional) - Name of this message or event on the wire.
• strict (optional) - Set true if enforcing value type for all attributes.
• attribute - An attribute in the input payload. These could also be known as fields, properties, keys.

Attributes

The top-level input payload is assumed to be a map, since this is common in many cases. If it is not in your case, transform the input into a map before validating it with Speck.

Attributes consist of a name, type, and optional validation arguments:

attribute <name>, <type>, <options>

Types:

• any
• boolean
• integer
• float
• string
• atom
• date (ISO 8601)
• time (ISO 8601)
• datetime (ISO 8601)
• map

Lists:

• Create a list of any type by wrapping the type in square brackets: [string]

Options:

• strict - Set true if enforcing value type instead of best attempt to coerce.
  • error - wrong_type
• default - The default value is used if the value is not present in the input.
• optional - Set true if a value for the attribute is not required.
  • error - not_present
• min - Minimum value for a number, or minimum length for a string.
  • error - less_than_min
• max - Maximum value for a number, or maximum length for a string.
  • error - greater_than_max
• length - Exact length for a string.
  • error - wrong_length
• values - List of valid values (an enum).
  • error - invalid_value
• format - Regular expression for the valid format of a string.
  • error - wrong_format

The any type:

• Intended to be a passthrough attribute where the value is not validated or coerced but the value must be not nil at a minimum if optional: false. The any type is useful if you want to split out schemas where the top level schema is a generic envelope containing a nested data structure that could possibly be of a polymorphic nature. Example:

struct MQTT.AWS.Shadow.Update.V1

name "aws_shadow_update"

attribute :state do
  attribute :desired, :any
end

struct MQTT.Light.State.V1

name "light_state"

attribute [:schedule] do
  attribute :start, :datetime
  attribute :stop,  :datetime
end

with {:ok, shadow}      <- Speck.validate(MQTT.AWS.Shadow.Update.V1, payload),
     {:ok, light_state} <- Speck.validate(MQTT.Light.State.V1, shadow.state.desired) do
  # do something with light_state
end

Examples

Example schemas can be found in /protocol.

Order of operations

Speck is designed to allow developers to focus on the data coming out of Speck rather than going into Speck. Therefore, its default behavior is to attempt to coerce a value to the attribute's type and then validate the value. This is known as permissive validation, and can help when working with unruly third-party protocols.

Strict validation

In some cases, strict validation may be required rather than permissive validation. Enabling Speck's strict validation will require the input values to match their spec's attribute type or else the payload will not be valid.

There are two ways to enable strict validation: globally, or per attribute.

Global

Add the top-level strict true property. Opt out per attribute.

struct MQTT.AddDevice.V1

name "add_device"

strict true

attribute :name, :string
attribute :type, :string, strict: false

Per attribute

Opt in per attribute by adding strict: true.

struct MQTT.AddDevice.V1

name "add_device"

attribute :id,   :integer, strict: true
attribute :name, :string