Property-Based Testing: Beyond Unit Tests in Elixir

#The Limits of Example-Based Tests

Traditional unit tests verify behavior for specific inputs: “given input X, expect output Y.” This works well for known edge cases but leaves vast regions of the input space unexplored. Property-based testing inverts the approach: define properties that must hold for all valid inputs, then let the framework generate hundreds of random inputs to verify them.

The Prismatic Platform’s TACH doctrine mandates property-based tests for all pure, stateless modules – validators, parsers, encoders, and type constructors.

#ExUnitProperties Basics

Elixir’s property testing library is ExUnitProperties, backed by StreamData for data generation:

defmodule PrismaticDD.Scoring.ValidatorPropertyTest do
  use ExUnit.Case, async: true
  use ExUnitProperties

  property "confidence scores are always between 0.0 and 1.0" do
    check all score <- float(min: 0.0, max: 1.0),
              label <- string(:alphanumeric, min_length: 1) do
      result = Validator.validate_confidence(%{score: score, label: label})
      assert {:ok, validated} = result
      assert validated.score >= 0.0
      assert validated.score <= 1.0
    end
  end
end

The check all macro generates random values matching the generators and runs the block for each combination. By default, it runs 100 iterations, configurable via max_runs.

#Writing Good Generators

StreamData provides primitive generators that compose into complex structures:

# Primitive generators
integer()                    # any integer
float(min: 0.0, max: 1.0)  # bounded float
string(:alphanumeric)        # alphanumeric string
binary()                     # random binary data

# Composite generators
list_of(integer())           # list of integers
map_of(atom(:alphanumeric), string(:utf8))  # map with atom keys

# Custom generators for domain types
def entity_generator do
  gen all name <- string(:alphanumeric, min_length: 1, max_length: 100),
          type <- member_of([:person, :company, :domain, :address]),
          confidence <- float(min: 0.0, max: 1.0) do
    %{name: name, type: type, confidence: confidence}
  end
end

The gen all macro (from StreamData) creates a generator that produces maps matching your domain model. These generators are reusable across tests.

#Properties Worth Testing

#Roundtrip Properties

Encoding followed by decoding should return the original value:

property "JSON roundtrip preserves data" do
  check all data <- map_of(string(:alphanumeric), string(:utf8)) do
    assert data == data |> Jason.encode!() |> Jason.decode!()
  end
end

#Idempotency Properties

Applying an operation twice should produce the same result as applying it once:

property "normalizing a slug is idempotent" do
  check all input <- string(:alphanumeric, min_length: 1) do
    once = Slug.normalize(input)
    twice = Slug.normalize(once)
    assert once == twice
  end
end

#Invariant Properties

Certain conditions must always hold regardless of input:

property "validated entities always have a non-empty name" do
  check all entity <- entity_generator() do
    case Validator.validate(entity) do
      {:ok, validated} -> assert String.length(validated.name) > 0
      {:error, _} -> :ok
    end
  end
end

#Shrinking: Finding Minimal Failing Cases

When a property test fails, StreamData automatically shrinks the failing input to the smallest value that still triggers the failure. This makes debugging dramatically easier.

For example, if a parser fails on a 200-character string, shrinking might reduce it to a 3-character string that exposes the same bug. Custom generators inherit shrinking behavior from their component generators.

#TACH Doctrine Integration

The TACH doctrine classifies modules by their testing requirements:

Pure modules are identified by having no side effects: no database calls, no PubSub broadcasts, no file I/O. These are the best candidates for property testing because they are deterministic and fast.

#Performance Considerations

Property tests run 100 iterations by default. For computationally expensive properties, reduce iterations:

property "expensive validation holds", max_runs: 25 do
  check all input <- complex_generator() do
    assert expensive_validation(input)
  end
end

For CI, the platform runs property tests with max_runs: 200 to increase confidence. Local development uses the default 100 for faster feedback.

#Real Impact

Property-based testing has caught bugs in the Prismatic Platform that example-based tests missed entirely: Unicode normalization edge cases in slug generation, floating-point precision issues in confidence score aggregation, and off-by-one errors in pagination boundary calculations. These bugs would have reached production without property testing.