BrainOS

Persistent, modular AI research system that runs unattended and produces rigorous, reproducible, paper-ready computer-science research.

The normative architectural source of truth is indexed in specification/README.md; operational explanations are indexed in docs/README.md; CLAUDE.md holds the provider-neutral engineering contract every agent working in this repository follows. This file is a practical overview and setup guide, not a changelog — detailed incident history lives in git log (this repo's commit messages are written as full narratives on purpose) and open work lives in TODO.md.

What BrainOS actually does

A resident autonomy daemon runs continuously, unattended, on this host. Its approved mission is rigorous, reproducible, paper-ready work for reputable IT conferences — not generic activity across unrelated domains. It maintains a hierarchical problem tree (decompose a research question into smaller, scored candidates; materialize the smallest ones into implementation tasks; propagate completion back up), drains that tree into coding jobs, and closes the loop by writing, reviewing, and rendering papers from the resulting evidence.

Infrastructure

• PostgreSQL with pgvector
• Redis
• host Ollama (serves the resident local models)
• SearXNG (self-hosted metasearch, used for local web lookups — see "Local web search" below)
• Python BrainOS application: FastAPI + Typer CLI, deployed via Docker Compose (api, scheduler, worker, matrix services, plus postgres/ redis/searxng)

Local generative models: one shared pool, not a fixed split

Every local generative role — planning, Cortex, research synthesis, classification, local code review, owner chat, and workspace implementation — runs through Codex OSS (codex exec --oss --local-provider ollama) against one of two resident models. Both currently-installed residents (qwen3.8-coding:27b and muse-glimmer-coding:30b-q4_K_M — check brain model-role-list rather than trusting a name here, since installed tags change) report the same capability set (vision, tool use, reasoning, structured output) and are interchangeable for essentially every role.

Role assignments in model_roles seed pool membership; they do not partition runtime work. Every call that needs a model resolves through model_registry.resolve_idle_role_model, which balances requests across every capable resident with a Redis-backed, cross-process round robin — the assigned model is the fallback/tiebreak, not an exclusive owner. This applies uniformly across the codebase: planning, the generic Supervisor's planner/reviewer/researcher cycle, opportunity discovery, paper drafting and worthiness assessment, memory consolidation, and the manual research CLI/API all share the same pool. The one place that deliberately does not use per-call balancing is the coding-cycle worker pool, which instead pins one long-running worker to each resident up front (genuine parallelism, not per-call selection) so two workers never race for the same lease.

Which model a call gets is a different question from how many things can run at once. The autonomy daemon runs BRAINOS_AUTONOMY_PARALLEL_LANES (default 2, matching the two resident models) concurrent lanes, each independently draining the generic Supervisor task cycle and the problem tree. Two previously separate mechanisms used to serialize this to exactly one lane system-wide regardless of available model capacity or how much independent work existed: AutonomousLoop.begin_tick refused a second AutonomyRun while any one was already "running" anywhere, and Supervisor.run_once held one unparameterized global Redis lock that made a second concurrent cycle return "busy" immediately even for a completely unrelated task. Both were removed in favor of the row-level locking (SELECT ... FOR UPDATE SKIP LOCKED) the coding lanes already prove safe for exactly this pattern; ProblemTreeService additionally takes a Postgres transaction-scoped advisory lock per module before decomposing or materializing it, so two lanes can never double-process the same one.

There is no BrainOS-native agent, command executor, controller tool protocol, or generative /api/chat fallback (the raw-Ollama branch in OllamaClient exists only as a narrow provider-protocol test seam and is never reached in production). BrainOS assembles role-relevant durable state into the prompt or job bundle; Codex itself owns reasoning, workspace inspection, shell commands, editing, compaction, and web search. BrainOS does not gate which of Codex's own tools it uses on any call, local or hosted.

Local web search

Codex's own --search flag exposes a web_search tool that, under --oss --local-provider ollama, routes through Ollama's own /api/experimental/web_search proxy — which forwards to Ollama Cloud and requires a paid/free account (OLLAMA_API_KEY) BrainOS does not configure. Rather than gate that tool off, every local generative role's prompt points the model at BrainOS's own free, local, credential-free SearXNG instance (BRAINOS_SEARXNG_URL, default http://127.0.0.1:18888) via an ordinary curl — Codex's own native shell tool, not a separate BrainOS mechanism. google cse and the plain wikipedia engine are the verified-working default engines; duckduckgo, brave, startpage, and wikidata are disabled in brain/docker/searxng/settings.yml because this host's shared institutional proxy IP gets CAPTCHA'd or rate-limited by all four.

Local setup

Prerequisites: Docker with Compose, uv, and an existing Ollama service bound to the host with nomic-embed-text and both configured generative role models available. BrainOS discovers and invokes Ollama models; model installation and root-owned Ollama startup parameters remain owner operations (CLAUDE.md, "Allowed scope").

cd /opt/brain
make setup

Edit .env and replace the local PostgreSQL password in both POSTGRES_PASSWORD and the host-side BRAINOS_DATABASE_URL. Compose overrides the application connection hosts internally. Never commit .env.

Start PostgreSQL and Redis, migrate, then run the API locally:

docker compose up -d postgres redis
make migrate
make run

The local API listens only on 127.0.0.1:8000. In another terminal:

make health
uv run brain health

To run the API in Compose instead (also localhost-only):

docker compose up -d --build postgres redis api
curl --fail --silent http://127.0.0.1:18000/health

Database and CLI operations

Apply migrations:

make migrate

Create a project and retain the printed UUID:

uv run brain project-create "Research project" --description "Initial scope"

Create a task with that project UUID, retain its UUID, and move it from proposed to ready through the audited transition endpoint:

uv run brain task-create PROJECT_UUID "Investigate a claim" --description "Collect evidence"
curl --fail --silent -X POST http://127.0.0.1:8000/tasks/TASK_UUID/transitions \
  -H 'content-type: application/json' \
  -d '{"state":"ready","actor":"owner"}'
uv run brain supervisor-cycle

The manual HTTP equivalent is:

curl --fail --silent -X POST http://127.0.0.1:8000/supervisor/cycles

Interactive API documentation is at http://127.0.0.1:8000/docs.

Quality checks

make format
make lint
make typecheck
make test

./scripts/install-git-hooks.sh   # enforce this gate on every commit, rebuild
                                  # containers automatically when brain/src changes

Docker builds copy the working tree, not git HEAD, so a change can end up running in production without ever being committed. Build application images with ./scripts/docker-build.sh (used automatically by the post-commit hook, and by hand for a manual rebuild), not a bare docker compose build: it bakes the exact git commit and a brain/src dirty-file count into the image, readable at GET /health's build field and logged at every service's startup. The post-commit hook also warns immediately if brain/src is still dirty right after a commit.

Run the opt-in PostgreSQL/pgvector integration test after exporting a URL that is reachable from the host:

export BRAINOS_TEST_DATABASE_URL='postgresql+psycopg://brain:YOUR_PASSWORD@127.0.0.1:15432/brain'
uv run pytest -m integration -q

Configuration

All application variables use the BRAINOS_ prefix. Important values are BRAINOS_DATABASE_URL, BRAINOS_REDIS_URL, BRAINOS_OLLAMA_URL, BRAINOS_SEARXNG_URL, BRAINOS_CHAT_MODEL, BRAINOS_EMBEDDING_MODEL, BRAINOS_OLLAMA_CHAT_KEEP_ALIVE_SECONDS, BRAINOS_OLLAMA_EMBEDDING_KEEP_ALIVE_SECONDS, BRAINOS_LOG_LEVEL, BRAINOS_SUPERVISOR_MAX_ACTIONS, BRAINOS_SUPERVISOR_MAX_RETRIES, BRAINOS_LOCAL_MODEL_IDLE_FAILOVER (default on — see "one shared pool" above), BRAINOS_AUTONOMY_PARALLEL_LANES (default 2, matching the two resident models — how many concurrent autonomy lanes run at once), BRAINOS_PROBLEM_TREE_DECOMPOSE_TIMEOUT_SECONDS (default 2700, 45 minutes), BRAINOS_PROBLEM_TREE_DECOMPOSE_HOSTED_ESCALATION_ENABLED (default on — escalate planning to a hosted agent after 3 consecutive local stalls), and BRAINOS_CODING_REPAIR_MAX_ATTEMPTS (default 6 — hard ceiling on total local-plus-repair attempts before a job is permanently retired instead of re-offered to every future repair pass).

On this host the two generative models use chat keep-alive -1 (resident until explicitly unloaded) and embeddings use 0 (unload immediately). The root-owned Ollama service must allow three loaded slots: two generative models plus the transient embedding model. This does not authorize a third generative model; measured VRAM permits only two.

General web search additionally requires BRAINOS_RESEARCH_PROVIDER and BRAINOS_RESEARCH_BASE_URL for the cited-research pipeline; credentials belong only in the untracked .env. Supported JSON adapters cover Brave, Tavily, Serper, Bing-compatible and SearXNG services. Academic adapters support arXiv, Crossref, OpenAlex, Semantic Scholar and PubMed metadata without paid test dependencies. This is separate from the local generative web-search path described above, which always uses the local SearXNG instance via Codex's own shell tool.

The schema fixes embedding columns at 768 dimensions for the configured nomic-embed-text model. Every Ollama embedding response is checked before it can be inserted; changing models therefore requires an explicit dimensional migration and matching BRAINOS_EMBEDDING_DIMENSIONS value.

Commands

# Cited general web research (after provider configuration)
uv run brain research "What evidence addresses this question?"

# Academic metadata and local PDF ingestion
uv run brain academic-search "retrieval augmented generation" --provider crossref
uv run brain pdf-ingest /tmp/brainos-artifacts/paper.pdf --render-directory /tmp/brainos-artifacts/pages

# Registry and deterministic routing
uv run brain models --refresh
uv run brain model-route vision
uv run brain model-role-list

# Build and use the network-disabled, non-root sandbox
make sandbox-build
printf 'print(6 * 7)\n' >/tmp/example.py
uv run brain python-run /tmp/example.py

# One-shot workflows
uv run brain sleep-run manual-2026-07-26
uv run brain model-discover
uv run brain benchmark MODEL_UUID
uv run brain strategic-review
uv run brain tuning-export /tmp/brainos-artifacts/tuning.jsonl

# Core scheduled workflows (see below for installers)
./scripts/run-brain.sh brain coding-cycle          # active mode: drain resident-model lanes
./scripts/run-brain.sh brain sleep-review-cycle    # maintenance + independent host-agent review
./scripts/run-brain.sh brain host-coding-repair-cycle # exhausted/self-improvement jobs
./scripts/run-brain.sh brain improvement-cycle     # host-agent review pass
./scripts/run-brain.sh brain problem-scan          # decompose the mission tree further
./scripts/run-brain.sh brain paper-writing-cycle   # readiness, draft, repair/review, render

# Everyday operational visibility
./scripts/run-brain.sh brain status                # task/coding-job/queue snapshot
./scripts/run-brain.sh brain doctor                # environment and dependency checks
./scripts/run-brain.sh brain autonomy              # inspect/control the autonomous worker
./scripts/run-brain.sh brain scheduler             # inspect/control the persistent scheduler
./scripts/run-brain.sh brain service               # manage installed systemd units
./scripts/run-brain.sh brain backup                # verified backup operations
./scripts/run-brain.sh brain workspace-cleanup     # preview reconstructible cache cleanup
./scripts/run-brain.sh brain workspace-cleanup --apply # preserve source/results/Git history
./scripts/run-brain.sh brain work-submit / work-retry # owner-originated coding jobs
./scripts/run-brain.sh brain mission --adopt-default --owner owner # adopt mission
./scripts/run-brain.sh brain mission / project-create / task-create # manual seeding

Run ./scripts/run-brain.sh brain --help for the complete, current command list; the ones above are the ones most relevant to understanding how the system actually runs day to day, not a full reference. The corresponding APIs are POST /research/runs, POST /research/academic, POST /research/pdf, GET /models, POST /models/refresh/ollama, POST /models/route, and POST /sleep/runs. See specification/RESEARCH.md, specification/MODELS.md, and specification/OPERATIONS_RUN2.md.

POSTGRES_* is the production credential source of truth. Compose and scripts/run-brain.sh construct the application URL without printing it; this keeps API, workers, migrations, CLI, and the initialized volume consistent.

See the autonomous purpose loop, operations, recovery, security, architecture, and policies for the full mechanics of each capability layer.

Matrix owner interface

Matrix is the deployed owner transport on this host. It uses outbound HTTPS /sync long polling and opens no inbound port. Only the configured owner user ID in the configured private room is accepted, and the bot ignores its own events. Telegram remains implemented for compatibility but is disabled and its Compose profile is not started. Free-text replies use a 220-token generation ceiling and a 1,200-character transport ceiling. Markdown is normalized to plain text, and unsolicited fenced/code-shaped output is withheld unless the owner explicitly asks for code.

See MATRIX_SETUP.md for the complete setup and test procedure.

Use /dashboard for the combined live snapshot. Additional structured commands include /workers, /events, /errors, /resources, /network, /proxy, /memory, /tools, /repositories, /improvement, and /needs-user. Ordinary chat uses durable owner conversation history, a verified runtime snapshot, and a curated durable-memory search relevant to the question, and answers directly rather than inventing status claims. Status answers about a specific task or job walk that task's problem-tree ancestry (title/statement/rationale at every level up to the mission) so "what are you doing and why" is answered from real stored planner output.

External status dashboard (static, pushed over SSH)

Separate from the Matrix /dashboard chat command above: dashboard.py's DashboardService renders a static HTML snapshot (problem-tree counts and top active modules, recent coding jobs with errors, each job's workspace top-level contents, and README/TODO/ARCHITECTURE rendered from Markdown) and pushes it via rsync over outbound SSH to an owner-controlled external webserver, refreshed by brainos-dashboard-push.timer every 20 minutes. This exists because api_host is hard-enforced to loopback (Settings.secure_configuration) and the institute firewall cannot be changed, so nothing on this host can be reached directly from outside it — the dashboard is pushed out instead of served, the same outbound-only shape Matrix already uses.

The canonical public dashboard and paper site is directly at https://simb.one/. It does not require BrainOS, a /brainOS path, or any other dashboard host in the URL. The configured SSH destination may still use the server-side directory /home/daniel/brainOS/; that filesystem path is an implementation detail and must not be presented as part of the public URL.

Configure via BRAINOS_DASHBOARD_PUSH_* in .env (see .env.example); BRAINOS_DASHBOARD_PUSH_ENABLED=false (the default) makes the CLI command and timer a no-op. Institute traffic must go through the mandatory HTTP(S) proxy (webproxy.berlin.ptb.de:8080), which also tunnels arbitrary CONNECT targets including raw SSH — BRAINOS_DASHBOARD_PUSH_SSH_PROXY_COMMAND carries that as an nc -X connect -x host:port %h %p ProxyCommand. Install or remove the timer with scripts/install-dashboard-push-timer.sh / scripts/rollback-dashboard-push-timer.sh; run it once by hand with brain render-and-push-dashboard.

Host coding and review agents

Several scheduled lanes do implementation and review work, differing in cost and capability, not in what CLAUDE.md permits: execution capability is the same for every tier, while rule 11 still requires approval for external side effects, and the no-root and Ollama/firewall/SSH restrictions bind all of them alike.

• Active mode (brainos-coding.timer, roughly every minute, coding-cycle) uses the resident local models exclusively through Codex OSS in YOLO mode, confined to each job's own disposable or shared-project workspace, one worker pinned per resident model. It excludes any job whose requested_by starts with improvement-cycle: — a local model never edits BrainOS's own source. A local reviewer model independently checks each job against a real, attributed pytest run before it can be marked complete. A job that exhausts BRAINOS_CODING_MAX_ATTEMPTS becomes failed; once total attempts (local plus any hosted repair) reach BRAINOS_CODING_REPAIR_MAX_ATTEMPTS, it is permanently retired instead of being offered to another repair pass.
• Sleep mode (brainos-review.timer, twice daily) runs a frontier hosted agent CLI — Claude Code first, with Codex fallback — with real write, edit, and shell access to /opt/brain, so it can fix a critical bug it finds directly (edit, run the quality gate, commit). It owns both architecture assessment and the improvement plan; no local model subsequently plans BrainOS architecture.
• Hosted repair (brainos-host-repair.timer, no more often than every six hours, at most three hosted CLI calls per cycle) lets a hosted agent attempt jobs whose local attempts were exhausted or whose review rejected them, plus self-improvement jobs the active-mode admission query excludes outright.
• Triage (brainos-codex-triage.timer, four times daily) runs Codex against the live problem tree and coding-job workspaces — never BrainOS's own architecture — sanity-checking modules, retiring dead branches, growing or concluding tree territory, and repairing real bugs including broken citations inside workspace documentation. A duplicate Claude triage timer exists but is disabled by default.
• Scientific publication (brainos-paper.timer, change-gated, default every 30 minutes, PaperWritingService) anchors on a current-mission depth-2 (occasionally depth-3) problem-tree module, requires real evidence and result artifacts before drafting. By default Claude assesses readiness and supplies a structure plan; only a positive verdict admits Astra to write, create evidence-grounded figures, repair, and self-check the paper in one session. There is no third independent model review. Author self-assessment is labeled honestly, and citation, evidence, and 15–20-page PDF gates remain. Failed attempts count toward the two-call cap; stages do not silently switch providers. Explicit legacy three-stage mode remains configurable. A rejection creates a durable PublicationGap (continue grows the missing work, pivot replaces the framing, stop supersedes it); a single focus slot boosts one candidate ahead of unrelated breadth. Rendering happens on this lane specifically because it is the only scheduled lane that runs on the host rather than in a container, and coding_workspace_root is a host path the containers do not mount.

Every hosted CLI subprocess (local or hosted) receives real database credentials (BRAINOS_DATABASE_URL, POSTGRES_*) so it can read and write BrainOS's own durable memory directly (brain memory-record, or a query), and — as a narrow, owner-approved exception — read-only system journal access (journalctl) once the owner has added the execution user to the systemd-journal group (see CLAUDE.md rule 8). Owner-interface/chat tokens remain excluded regardless of tier.

Every install/rollback script above also runs scripts/ensure-linger.sh first (loginctl enable-linger), so these brainos-*.timer units keep running across logout instead of existing only while a login session is open.

Enterprise HTTPS proxies

BrainOS can add organization-managed CA roots to every Python HTTPS client without replacing the normal trust store. Place PEM certificates or bundles in the Git-ignored certs/ directory, or set BRAINOS_EXTRA_CA_CERT to a PEM path inside that read-only mount, then recreate the application containers. With no extra certificates configured, startup is a no-op and normal public PKI trust is unchanged.

See the enterprise proxy deployment guide and implementation report. TLS verification must never be disabled.

Production start

cd /opt/brain
cp -n .env.example .env
# Set POSTGRES_PASSWORD and the owner-only BRAINOS_MATRIX_* values in .env.
# Leave BRAINOS_TELEGRAM_ENABLED=false.
make setup
docker compose up -d postgres redis searxng
make migrate
./scripts/run-brain.sh brain mission --adopt-default --owner owner
./scripts/run-brain.sh brain scheduler --initialize
./scripts/docker-build.sh api scheduler worker matrix
docker compose --profile matrix up -d api scheduler worker matrix
curl --fail --silent http://127.0.0.1:18000/ready

If the package index is temporarily unreachable but a previously verified BrainOS image is present, use the documented offline overlay procedure in docs/OPERATIONS.md; it replaces application source only and explicitly reports optional dependency degradation.