Architecture

Components

• Core Warehouse (backend/) — Node.js/TypeScript + Express REST API under /api/v1/. Owns relational metadata (PostgreSQL via Prisma), publishes/consumes lifecycle events (RabbitMQ), and hosts the Pipeline Router that dispatches steps, handles retries/backoff, and sweeps for step timeouts.
• Plugin Workers (plugins/*/) — one independent Docker container per plugin. Each subscribes to the event broker for its own routing key and reports results via a single internal HTTP callback.
• Web UI (frontend/) — React + Vite + Tailwind SPA, three primary sections (Resources, Pipelines, Plugins) plus a global search, served via nginx which also reverse-proxies /api and /registry to their respective backend services.
• Plugin Registry (registry/ + community-site/) — registry is a standalone service (its own SQLite-backed API, with user accounts) that any DataCore deployment's frontend can browse or post to; community-site is a standalone website that fronts the same API as its own destination, independent of the rest of the stack. Hosts two kinds of public listings: plugin listings (metadata + a link to the plugin's own repo/Docker image — not a code-hosting or auto-deploy system) and DataCore Bundles (a shareable list of resource definitions + the pipeline that processed them, for reprocessing elsewhere — see Sharing & Importing Bundles). Posting either requires a verified account; browsing doesn't.
• Storage tiers: PostgreSQL (relational metadata only), MinIO (raw files + text/JSON artifacts, S3-compatible), and whatever store a plugin's artifacts belong in (e.g. Qdrant for vectors) — Postgres never stores file content or vectors directly, only reference pointers.

Guiding principles

These come from the project constitution (.specify/memory/constitution.md) and shape every design decision:

1. Event-Driven Architecture — all state transitions are broker events; no synchronous cross-service calls except the one explicitly-allowed internal artifact callback.
2. Plugin Isolation — each plugin is its own container; a crash, hang, or bug in one plugin cannot affect Core or any other plugin.
3. Schema-First Data Models — Resource/Pipeline/Plugin/Artifact schemas are defined before any endpoint that reads or writes them.
4. API Contract Discipline — every REST endpoint lives under /api/v1/ with one consistent JSON error envelope ({ "error": { "code", "message" } }).
5. Separation of Storage — relational metadata, raw files, and generated artifacts each live in their own tier; nothing is duplicated across tiers.
6. UI Consistency — the Web UI reflects live API state; no hardcoded mock data once a view is wired up.
7. Test Discipline — every REST endpoint and event handler has an integration test run against real Postgres/RabbitMQ/MinIO/Qdrant — nothing is mocked.
8. No Silent Failures — a failed step surfaces a specific failure_reason on the resource; nothing is left Processing forever without either completing, failing, or (for a non-responding step) being caught by the timeout sweep.

Request flow: registering a resource

User → POST /api/v1/resources
     → Core persists Resource (PENDING), matches a Pipeline by trigger_type
     → Core publishes RESOURCE_CREATED
     → Pipeline Router consumes it, snapshots the pipeline's steps onto the resource, dispatches step 0
     → Plugin Worker consumes PIPELINE_STEP_DISPATCHED, does its work
     → Plugin calls back POST /api/v1/internal/artifacts/{resource_id}
     → Core upserts the Artifact, advances to the next step (or marks COMPLETED)

Full detail, including the retry/backoff/timeout state machine, is in Events reference and contracts/events.md in the spec.