Migrations and upgrades¶

The plugin ledger is a new additive SurrealKit schema table. Existing source_package rows have no plugin owner or generation and remain valid legacy single-source installs; they keep their component digest, activation state, credentials and mining cursors without a data backfill. Installing a plugin never silently takes ownership of an existing legacy source with the same ID. If a legacy source was removed after mining, plugin install --adopt-source <id> is the explicit handoff of its preserved cursor and drawers to the plugin supplying that same source kind. When a Core release stops bundling a source, its stored enabled-state row remains. After upgrading Core, remove the old source package row with memcastle source remove codex or memcastle source remove chatgpt before installing the OpenAI plugin. This clears the source's enabled state and credential, but not its miner, mined documents or cursor. Review the new permissions, install with --adopt-source for each source with mined history, enable it again and sign ChatGPT in again before web mining. Mined drawers and source cursors live in separate tables and survive plugin updates and uninstall.

MemCastle keeps its data in a versioned shape. When a release changes that shape, it ships a migration, and the daemon applies it before it serves anyone. For most upgrades you do nothing: install the new version and start it.

What gets migrated¶

Two kinds of change are tracked separately:

• Schema. The database's tables, fields and indexes, declared in files embedded in the binary. They are synchronized automatically by SurrealKit on every start. This is idempotent, so an unchanged schema costs nothing.
• Data. Ordered, one-time steps for changes that cannot be expressed as a schema sync, such as reshaping stored records. MemCastle records how far a palace has got in a version number, and each step runs exactly once.

Version 3 is made of these pieces, run in this order:

• Renamed fields. Moves values stored under a name this release changed to the new name. An audit job's scope becomes wing, in the job and in its report, so a queued audit keeps its filter. The adapter's provider becomes source in mining jobs and their reports, in each drawer's origin (and the origin copied onto extracted facts) and on the source table, and the mined place's own id, which was called source, becomes source_id.
• Deduplication keys. Fills the derived keys deduplication reads (a drawer's fingerprint, an entity's key and aliases) on records written earlier. It merges nothing, so existing duplicates stay as they are.
• Supersession lineage. Links a drawer closed before supersession links existed to the successor it was replaced by, when exactly one drawer qualifies (see ADR-032). An ambiguous or unmatched close is left alone.

Adding the source and source_document tables (Mining sources) needed no data migration: they are schema, created empty by the next start, and nothing existing is reshaped.

A fresh palace starts at version 0 and is brought to the latest version on its first start. Data migrations never delete canonical memory, and released migrations are never edited: a mistake in one is fixed by a new migration.

What happens on start¶

sequenceDiagram
    participant R as Migration runner
    participant DB as Palace database

    R->>DB: sync schema (creates the version table if missing)
    R->>DB: take the migration lock
    R->>DB: read the current data version
    loop each pending step, in order
        R->>DB: run the step
        R->>DB: record the new version
    end
    opt at least one step ran
        R->>DB: sync schema again (picks up anything else the release ships)
    end
    R->>DB: release the lock

When no step is pending, the schema is not synced a second time: the first sync has already made it current.

memcastle serve runs this before it starts answering requests, so an MCP client never sees a half-migrated palace. The listener is already bound by then, and connections wait in its backlog. If a step fails, the daemon exits with memcastle::migrate::failed instead of serving. The version stays at the last step that succeeded, so once the cause is fixed a later start resumes rather than starts over. The lock is a lease, so a crashed run does not block the next one forever; a start that finds it held fails with memcastle::migrate::locked.

Existing embeddings without provenance¶

Version 4 records the switch to profile-scoped vectors after SurrealKit adds the derived-vector tables, and retires the unscoped legacy HNSW index with SurrealKit's destructive rollout API. The old drawer.embedding values remain recoverable, but their provider and model cannot be inferred reliably; semantic search does not use them in a selected profile. Migration does not contact a provider or rewrite canonical content. After configuring a profile, run memcastle embed --profile NAME to fill its missing vectors as a durable job. Use --refresh stale or --refresh all only when those replacements are intended.

Inspect or run migrations by hand¶

memcastle migrate runs the same code as serve, but talks to storage directly and needs no daemon.

memcastle migrate --status   # report, change nothing
memcastle migrate --check    # like --status, but fail if anything is pending
memcastle migrate            # apply whatever is pending

Both report flags print JSON when piped, or with --json, and a short summary in a terminal:

{
  "current_version": 3,
  "latest_version": 3,
  "pending": []
}

--check exits with an error (memcastle::migrate::pending) when pending is not empty, so it fits in a CI job or a pre-start script. Applying prints what happened, in the same two forms:

{
  "from_version": 3,
  "to_version": 3,
  "applied": []
}

An embedded palace can only be opened by one process, so stop the daemon before running memcastle migrate; otherwise it fails with memcastle::store::backend_failed. While a daemon is running, memcastle status shows the same information under its datastore line (migrations 4/4), and lists any pending steps.

Upgrading MemCastle¶

1. Stop the daemon: memcastle daemon stop.
2. Back up the palace directory if the data matters, see Storage and data.
3. Install the new version, see Installation.
4. Start it: memcastle serve. Pending migrations are applied first, and the daemon starts serving once they succeed.
5. Confirm with memcastle status: the datastore line should read ok, with equal current and latest versions.

Running jobs are safe across the restart: they checkpoint on shutdown and resume on the next start.

The 0.4.1 registry move¶

The official source registry moves from https://noirbizarre.github.io/memcastle/registry.json to https://memcastle.github.io/registry.json with 0.4.1. An installation using the default mining.registries picks up the new location when the daemon restarts. If you explicitly configured the old URL under [mining], replace it with the new URL in your configuration before restarting. There is no database migration for this change.

A source previously installed from the old registry keeps its original registry URL as its update origin. To switch that source to the new origin, install the same name again from the new registry:

memcastle source install <name> --registry https://memcastle.github.io/registry.json
memcastle source update <name> --check

The install prompts for the source's permissions again, replaces the package in place, preserves its enabled state, and records the new registry as its origin; it does not remove stored memories or the source's sign-in. Sources that shipped with MemCastle are bundled and update with the MemCastle release instead. Do not rely on a redirect from the old GitHub Pages URL: whether it stays reachable after repository transfer must be checked independently; 0.4.0 users need 0.4.1 (or an explicit working registry override) to discover new sources.

Design¶

The reasoning — why schema and data are separate, why migrations are forward-only, and why the daemon fails closed — is recorded in ADR-004.