Production-Grade Data Platform for 211 Social Services

1. What is a 211 Social Services Data Platform?

A 211 social services data platform is the technical layer behind 211 helplines and search portals that connect people in need with local resources: food banks, shelters, mental health providers, utility assistance, and similar services. The platform ingests resource data from independent organizations operating in different states, each on its own case management system (iCarol, WellSky, RTM, or similar), and exposes a unified, searchable interface to the public and to partner agencies.

Connect211 is a production deployment of this model across multiple U.S. states. The platform must answer a deceptively simple question: when someone searches for help right now, is the result correct, current, and trustworthy?

2. Initial State: Working Pipelines, Eroding Trust

Before the rewrite, the platform ran on pragmatic, pipeline-shaped data flows. Source data was fetched, normalized, and pushed into publishing stores through a sequence of tasks. It worked, organizations adopted it, and impact grew.

For a system intended to support people under real-world pressure, this state was unsustainable. The problem was not a lack of data or compute. It was a lack of structural guarantees about what the platform was producing.

3. Project Objectives

The goal was not a new dashboard. It was to rebuild the platform so that reliability, freshness, and lineage were properties of the architecture, not of the team.

4. Target Architecture: Writers, Readers, and the Asset Graph

We redesigned the platform around a clear separation between ingestion (writers) and publishing (readers), with a shared system of record underneath and asset-based orchestration coordinating everything.

5. Source Heterogeneity as the Core Constraint

The hardest constraint was not throughput or storage. It was heterogeneity. Each data provider differed along several axes simultaneously, and those differences dominated the design.

Treating ingestion as a uniform, pipeline-shaped process led to brittle assumptions and cross-tenant coupling. The system only became manageable once heterogeneity was treated as fundamental rather than incidental. Every writer is a separate project, on its own schedule, with its own connector adapter.

6. HSDS Normalization as an Architectural Contract

Normalization into an HSDS-style model was not implemented as a downstream convenience. It became an architectural contract that all downstream consumers rely on, whether they know it or not.

Because every reader depends on this contract, normalization cannot be best-effort or deferred to the end of a pipeline. The implementation enforces it through a strict layering:

• Raw source data is written verbatim into Snowflake source schemas (no interpretation)
• DBT ELT projects transform raw data into standardized intermediate models (the shared shape)
• DBT STAGE projects apply tenant-specific adaptations while preserving the contract (the shared semantics)

This separation makes it explicit where interpretation happens. When a field is wrong in the normalized model, the question becomes which contract was violated, not what broke in the pipeline.

7. Transition to Asset-Based Orchestration

The shift to asset-based orchestration was driven less by tooling preference and more by a change in mental model. The question stopped being what jobs should run and started being what data must exist.

Dagster assets provided a way to encode those questions directly. A simplified example from a writer project shows how DBT models are treated as assets rather than opaque steps:

# writer-xyz/assets.py (excerpt)
from dagster_dbt import load_assets_from_dbt_project

dbt_elt_assets = load_assets_from_dbt_project(
    project_dir="dbt_elt",
    profiles_dir="dbt_elt",
)

This does not describe how DBT runs. It declares that the resulting tables are first-class assets with lineage and state. Once assets replaced jobs as the primary abstraction, freshness, lineage, and partial recomputation became explicit rather than implicit.

8. Partitioning for Failure Isolation

Partitioning was critical for isolating failures. We partitioned primarily along tenant and state boundaries rather than time, because operational reality showed that data issues almost always affected a single organization or region.

This had a side effect that mattered just as much: it made the cost of adding a new tenant predictable. New organizations could be onboarded without increasing shared operational risk.

9. Data Quality Embedded in the Asset Graph

Data validation moved into the asset graph itself. Instead of post-hoc checks, validations became explicit dependencies. If a validation asset failed, downstream assets simply did not materialize.

An example pattern used across writers:

@asset
def validate_staging_tables(staging_tables):
    assert staging_tables.count_missing_ids() == 0

The check itself is intentionally simple. The key point is not the assertion but that failure is structural. The system records that an expected artifact does not exist, rather than silently publishing bad data. This shifted failure detection earlier and reduced the blast radius of errors.

10. Operational Outcomes

Day-to-day operations changed in several concrete ways once the asset graph was in place.

• On-call work shifted from rerunning pipelines to inspecting asset lineage
• Partial backfills became routine rather than exceptional, scoped to a single tenant's partitions
• Publishing delays were easier to attribute to specific upstream causes because the graph shows what depended on what
• New tenants could be added without increasing shared operational risk
• Auditability improved because every artifact carries its provenance and every update has a recorded reason

None of this eliminated operational effort. It made that effort more focused and less reactive.

11. Open Trade-offs and Unresolved Questions

Some challenges remain unresolved and shape the next iteration of the platform.

With more time, the next investments are unified observability across the stack and more formal schema versioning across tenants.

12. Why These Lessons Matter Beyond This Platform

The lessons from Connect211 are not unique to social services data. Any platform operating in civic tech, govtech, environmental data, or public health shares similar constraints: multiple data producers, uneven quality, and real-world consequences for failure.

The core takeaway is not “use asset-based orchestration.” The takeaway is to treat data artifacts as obligations. Once that shift happens, many architectural decisions become clearer.

Frequently Asked Questions