ETS-Backed Registries: Sub-Millisecond Access in Elixir

#The Problem with Database-Backed Registries

When your platform manages 552 AI agents, 157 OSINT tool adapters, and hundreds of blog articles, every read matters. Querying PostgreSQL for metadata that rarely changes introduces unnecessary latency and database load. The solution is Erlang Term Storage (ETS), a built-in key-value store that lives in process memory and supports concurrent reads without locking.

#ETS Fundamentals for Registry Design

ETS tables are created by a process and destroyed when that process terminates. For registries, we use :named_table and :public access so any process can read without message passing:

:ets.new(:agent_registry, [
  :set,
  :named_table,
  :public,
  read_concurrency: true
])

The read_concurrency: true option optimizes for workloads where reads vastly outnumber writes, which is exactly the pattern for registries.

#Compile-Time Loading Pattern

The platform populates ETS tables at application startup by scanning the filesystem. The agent registry, for example, walks the .aiad/agents/ directory and parses each .agent.md file:

defp load_agents do
  Path.wildcard("#{root}/.aiad/agents/*.agent.md")
  |> Enum.map(&parse_agent_file/1)
  |> Enum.each(fn agent ->
    :ets.insert(:agent_registry, {agent.slug, agent})
  end)
end

This pattern means the registry is always consistent with the filesystem. Adding a new agent file automatically makes it discoverable at next startup.

#The Ensure-Table Pattern

A critical pattern for ETS registries is idempotent table creation. Multiple processes may attempt to access the registry before it is initialized:

def ensure_table do
  case :ets.info(@table) do
    :undefined ->
      :ets.new(@table, [:set, :named_table, :public, read_concurrency: true])
      load_data()
      :ok
    _ ->
      :ok
  end
end

Every public function calls ensure_table/0 before accessing data. This eliminates race conditions during startup without requiring a supervision tree dependency.

#Query Patterns

ETS supports pattern matching through :ets.match_object/2 and match specifications via :ets.select/2. For the OSINT tool registry, we filter by category:

def tools_by_category(category) do
  ensure_table()
  :ets.select(@table, [
    {{:"$1", :"$2"},
     [{:==, {:map_get, :category, :"$2"}, category}],
     [:"$2"]}
  ])
end

For simpler lookups, :ets.lookup/2 returns results in constant time:

def get_by_slug(slug) do
  ensure_table()
  case :ets.lookup(@table, slug) do
    [{^slug, article}] -> {:ok, article}
    [] -> {:error, :not_found}
  end
end

#Performance Characteristics

ETS reads are measured in microseconds, not milliseconds. In benchmarks on the Prismatic Platform:

Compare this to PostgreSQL queries that typically take 1-5ms even with connection pooling. For metadata registries, ETS provides a 1000x improvement.

#Table Lifecycle Management

ETS tables are owned by the process that creates them. If that process crashes, the table is destroyed. The platform uses two strategies to handle this:

1. Supervisor-owned tables: A dedicated GenServer creates and owns the table. The supervisor restarts it on crash, and ensure_table/0 repopulates data.

2. Heir tables: Using the :heir option, table ownership transfers to another process on crash, preserving data across restarts.

For registries with cheap reload costs (filesystem scanning), strategy 1 is simpler and preferred. For registries with expensive initialization, strategy 2 avoids data loss.

#When Not to Use ETS

ETS is not a database replacement. It lacks transactions, persistence across restarts, and distributed replication. Use ETS for:

• Read-heavy metadata registries
• Caching layers with known invalidation patterns
• Counters and rate limiting
• Lookup tables loaded at startup

Use PostgreSQL (via Ecto) for data that must survive restarts, requires ACID transactions, or needs complex querying beyond key-value patterns.