# JARVIS Knowledge Base (Phase 3) Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Replace the TODO-stub `POST/GET /api/v1/documents` endpoints with a real document-upload + semantic-search knowledge base, backed by pgvector in the existing Postgres instance and Ollama for embeddings.

**Architecture:** Documents are chunked (character-based, with overlap) on upload, each chunk is embedded via Ollama's `nomic-embed-text` model, and stored in a new `document_chunks` table (pgvector column). Search embeds the query the same way and ranks chunks by cosine distance (`<=>`).

**Tech Stack:** FastAPI, psycopg2 (existing `db_query` helper), pgvector Postgres extension, aiohttp (existing Ollama HTTP pattern), pytest (new, local-only dev dependency).

**Spec:** `docs/superpowers/specs/2026-09-12-knowledge-base-design.md`

## Global Constraints

- No dedicated git repository exists for this project yet (the working directory sits inside an unrelated, very large home-directory-level repo that must not be touched). **Every "Commit" step below is replaced by "confirm the file is saved" — do not run `git add`/`git commit`.**
- Runtime is the live VPS at `72.61.186.98`, container `jarvis-api` (image `python:3.11-slim`, code bind-mounted from `/home/jarvis-core/jarvis/api`, `uvicorn --reload`). SSH: `ssh -F /dev/null -o IdentitiesOnly=yes -i ~/.ssh/jarvis_core_key jarvis-core@72.61.186.98`. Local files live in `Claude outputs/` (note the space — quote the path in shell commands) and must be scp'd to `/home/jarvis-core/jarvis/...` to take effect.
- Reuse the existing `db_query(query, params, fetch)` async helper in `main.py` for all DB access — do not introduce an ORM.
- Follow the existing code style in `main.py`: section comments (`# ============ X ============`), try/except around endpoint bodies that logs and raises `HTTPException`, async endpoints.
- Embedding model: `nomic-embed-text` (768 dimensions) — this fixes the `vector(768)` column width used below.
- Chunking: ~1000 characters, 100 character overlap (from the spec) — do not change these defaults without updating the spec.

---

### Task 1: pgvector extension + `document_chunks` table

**Files:**
- Modify: `Claude outputs/docker-compose.yml` (postgres image)
- Create: `Claude outputs/migrations/002_document_chunks.sql`

**Interfaces:**
- Produces: Postgres table `document_chunks(id, document_id, chunk_index, content, embedding vector(768), created_at)`, used by Task 4 and Task 5's DB helpers.

**Context:** the plain `postgres:16-alpine` image does not ship the pgvector extension. `pgvector/pgvector:pg16` is a drop-in image (built from postgres:16) with the extension precompiled — swapping the image and recreating the container is safe, the existing `postgres_data` volume is untouched.

- [ ] **Step 1: Swap the Postgres image to the pgvector-enabled build**

Edit `Claude outputs/docker-compose.yml`:

```yaml
  postgres:
    image: pgvector/pgvector:pg16
```

(only the `image:` line under the `postgres:` service changes — everything else stays as-is).

- [ ] **Step 2: Deploy the compose change and recreate the postgres container**

```bash
SSHOPTS="-F /dev/null -o IdentitiesOnly=yes -i /c/Users/Jonny/.ssh/jarvis_core_key"
scp $SSHOPTS "C:\Users\Jonny\Projekte\Claude\JARVIS\Claude outputs\docker-compose.yml" jarvis-core@72.61.186.98:/home/jarvis-core/jarvis/docker-compose.yml
ssh $SSHOPTS jarvis-core@72.61.186.98 "cd /home/jarvis-core/jarvis && docker compose up -d postgres && sleep 5 && docker compose ps postgres"
```

Expected: `jarvis-postgres` shows `Up ... (healthy)` with the new image.

- [ ] **Step 3: Write the migration file**

Create `Claude outputs/migrations/002_document_chunks.sql`:

```sql
CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE document_chunks (
    id SERIAL PRIMARY KEY,
    document_id INTEGER NOT NULL REFERENCES documents(id) ON DELETE CASCADE,
    chunk_index INTEGER NOT NULL,
    content TEXT NOT NULL,
    embedding vector(768) NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

CREATE INDEX idx_document_chunks_document_id ON document_chunks(document_id);
CREATE INDEX idx_document_chunks_embedding ON document_chunks
    USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100);
```

- [ ] **Step 4: Apply the migration to the live database**

```bash
SSHOPTS="-F /dev/null -o IdentitiesOnly=yes -i /c/Users/Jonny/.ssh/jarvis_core_key"
scp $SSHOPTS "C:\Users\Jonny\Projekte\Claude\JARVIS\Claude outputs\migrations\002_document_chunks.sql" jarvis-core@72.61.186.98:/home/jarvis-core/jarvis/migrations/002_document_chunks.sql
ssh $SSHOPTS jarvis-core@72.61.186.98 "docker exec -i jarvis-postgres psql -U jarvis -d jarvis < /home/jarvis-core/jarvis/migrations/002_document_chunks.sql"
```

Expected output ends with `CREATE INDEX` (no errors). If `CREATE EXTENSION` fails with "could not open extension control file", Step 1/2 didn't take effect — re-check the image was actually swapped (`docker inspect jarvis-postgres | grep Image`).

- [ ] **Step 5: Verify**

```bash
ssh $SSHOPTS jarvis-core@72.61.186.98 "docker exec jarvis-postgres psql -U jarvis -d jarvis -c '\d document_chunks'"
```

Expected: column list showing `embedding | vector(768)` among others.

- [ ] **Step 6: Confirm files saved** (no git repo for this project — see Global Constraints)

---

### Task 2: Chunking function (TDD)

**Files:**
- Modify: `Claude outputs/main.py` (add `chunk_text`)
- Test: `Claude outputs/tests/test_chunking.py`
- Create: `Claude outputs/requirements-dev.txt`

**Interfaces:**
- Produces: `chunk_text(text: str, chunk_size: int = 1000, overlap: int = 100) -> list[str]` in `main.py`, consumed by Task 4's upload endpoint.

- [ ] **Step 1: Add pytest as a local-only dev dependency**

Create `Claude outputs/requirements-dev.txt`:

```
pytest==8.3.3
```

- [ ] **Step 2: Install it locally**

```bash
cd "C:\Users\Jonny\Projekte\Claude\JARVIS\Claude outputs"
python -m pip install -r requirements-dev.txt -r requirements.txt
```

- [ ] **Step 3: Write the failing tests**

Create `Claude outputs/tests/test_chunking.py`:

```python
import sys
import os

sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))

from main import chunk_text


def test_chunk_text_empty_returns_empty_list():
    assert chunk_text("") == []


def test_chunk_text_shorter_than_chunk_size_returns_single_chunk():
    assert chunk_text("hello", chunk_size=1000, overlap=100) == ["hello"]


def test_chunk_text_exact_multiple_of_chunk_size():
    text = "a" * 20
    chunks = chunk_text(text, chunk_size=10, overlap=2)
    assert chunks == [text[0:10], text[8:18], text[16:20]]
```

- [ ] **Step 4: Run tests to verify they fail**

```bash
cd "C:\Users\Jonny\Projekte\Claude\JARVIS\Claude outputs"
python -m pytest tests/test_chunking.py -v
```

Expected: FAIL with `ImportError: cannot import name 'chunk_text'` (or similar — `chunk_text` doesn't exist yet). Note: importing `main.py` will also try to import `psycopg2`, `redis`, `anthropic`, `aiohttp` — these must already be installed locally via Step 2 for the import to get far enough to fail on `chunk_text` specifically, not on a missing package.

- [ ] **Step 5: Implement `chunk_text`**

In `Claude outputs/main.py`, add near the other pure helpers (e.g. above `# ============ DB HELPERS ============`):

```python
def chunk_text(text: str, chunk_size: int = 1000, overlap: int = 100) -> list:
    if not text:
        return []
    if len(text) <= chunk_size:
        return [text]
    chunks = []
    step = chunk_size - overlap
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunks.append(text[start:end])
        if end >= len(text):
            break
        start += step
    return chunks
```

- [ ] **Step 6: Run tests to verify they pass**

```bash
cd "C:\Users\Jonny\Projekte\Claude\JARVIS\Claude outputs"
python -m pytest tests/test_chunking.py -v
```

Expected: 3 passed.

- [ ] **Step 7: Confirm files saved** (no git repo for this project — see Global Constraints)

---

### Task 3: Ollama embedding helper

**Files:**
- Modify: `Claude outputs/main.py` (add `OLLAMA_EMBED_MODEL` config + `get_embedding`)
- Modify: `Claude outputs/docker-compose.yml` (jarvis-api environment)

**Interfaces:**
- Consumes: `OLLAMA_HOST` (already defined in `main.py`), `aiohttp` (already imported).
- Produces: `async def get_embedding(text: str) -> list` in `main.py`, consumed by Task 4 and Task 5.

- [ ] **Step 1: Add the embed-model config constant**

In `Claude outputs/main.py`, next to the other `# ============ CONFIG ============` constants:

```python
OLLAMA_EMBED_MODEL = os.getenv("OLLAMA_EMBED_MODEL", "nomic-embed-text")
```

- [ ] **Step 2: Add `get_embedding`**

In `Claude outputs/main.py`, add after `chunk_text`:

```python
async def get_embedding(text: str) -> list:
    timeout = aiohttp.ClientTimeout(total=30)
    async with aiohttp.ClientSession(timeout=timeout) as session:
        async with session.post(
            f"{OLLAMA_HOST}/api/embeddings",
            json={"model": OLLAMA_EMBED_MODEL, "prompt": text},
        ) as resp:
            if resp.status != 200:
                body = await resp.text()
                raise RuntimeError(f"Ollama embedding failed ({resp.status}): {body}")
            data = await resp.json()
            return data["embedding"]
```

- [ ] **Step 3: Add `OLLAMA_EMBED_MODEL` to the deployed environment**

In `Claude outputs/docker-compose.yml`, under `jarvis-api: environment:`, add a line next to `OLLAMA_HOST`:

```yaml
      - OLLAMA_EMBED_MODEL=nomic-embed-text
```

- [ ] **Step 4: Confirm files saved** (no automated test here — `get_embedding` is verified end-to-end in Task 4; see Global Constraints for why there's no commit step)

---

### Task 4: Wire up document upload

**Files:**
- Modify: `Claude outputs/main.py` (`insert_document`, `insert_document_chunks`, `upload_document`)

**Interfaces:**
- Consumes: `chunk_text` (Task 2), `get_embedding` (Task 3), `db_query` (existing), `DEFAULT_USER_ID` (existing global).
- Produces: DB helpers `insert_document(title, content, document_type) -> int` and `insert_document_chunks(document_id: int, chunks: list) -> None`, where `chunks` is `list[tuple[str, list]]` of `(chunk_text, embedding)` pairs.

- [ ] **Step 1: Add a vector-literal formatting helper and the DB helpers**

In `Claude outputs/main.py`, add near the other DB helpers (after `save_message`):

```python
def _vector_literal(embedding: list) -> str:
    return "[" + ",".join(repr(float(x)) for x in embedding) + "]"


async def insert_document(title: str, content: str, document_type: str) -> int:
    row = await db_query(
        "INSERT INTO documents (user_id, title, content, document_type) VALUES (%s, %s, %s, %s) RETURNING id",
        (DEFAULT_USER_ID, title, content, document_type),
        fetch="one",
    )
    return row["id"]


async def insert_document_chunks(document_id: int, chunks: list):
    for index, (content, embedding) in enumerate(chunks):
        await db_query(
            """
            INSERT INTO document_chunks (document_id, chunk_index, content, embedding)
            VALUES (%s, %s, %s, %s::vector)
            """,
            (document_id, index, content, _vector_literal(embedding)),
        )
```

- [ ] **Step 2: Replace the `upload_document` endpoint**

In `Claude outputs/main.py`, replace the existing stub:

```python
@app.post("/api/v1/documents")
async def upload_document(title: str, content: str, document_type: str = "general"):
    """Upload document to knowledge base"""
    if not pg_pool:
        raise HTTPException(status_code=503, detail="Database is not configured")
    if not content.strip():
        raise HTTPException(status_code=422, detail="content must not be empty")

    try:
        logger.info(f"Uploading document: {title}")
        document_id = await insert_document(title, content, document_type)
        chunks = chunk_text(content)
        embedded_chunks = [(c, await get_embedding(c)) for c in chunks]
        await insert_document_chunks(document_id, embedded_chunks)

        return {
            "document_id": document_id,
            "title": title,
            "status": "indexed",
            "chunk_count": len(chunks),
        }
    except HTTPException:
        raise
    except Exception as e:
        logger.error(f"Document upload error: {str(e)}")
        raise HTTPException(status_code=503, detail=f"Embedding failed: {str(e)}")
```

- [ ] **Step 3: Deploy**

```bash
SSHOPTS="-F /dev/null -o IdentitiesOnly=yes -i /c/Users/Jonny/.ssh/jarvis_core_key"
OUT="C:\Users\Jonny\Projekte\Claude\JARVIS\Claude outputs"
scp $SSHOPTS "$OUT/main.py" jarvis-core@72.61.186.98:/home/jarvis-core/jarvis/api/main.py
scp $SSHOPTS "$OUT/docker-compose.yml" jarvis-core@72.61.186.98:/home/jarvis-core/jarvis/docker-compose.yml
ssh $SSHOPTS jarvis-core@72.61.186.98 "cd /home/jarvis-core/jarvis && docker compose up -d jarvis-api && sleep 8 && docker logs --tail 20 jarvis-api"
```

Expected: log ends with `Application startup complete.` (no traceback).

- [ ] **Step 4: Pull the embedding model**

```bash
ssh $SSHOPTS jarvis-core@72.61.186.98 "docker exec jarvis-ollama ollama pull nomic-embed-text"
```

Expected: ends with `success`.

- [ ] **Step 5: Test the upload endpoint against the live API**

```bash
ssh $SSHOPTS jarvis-core@72.61.186.98 "curl -s -X POST 'http://localhost:8000/api/v1/documents?title=Oeffnungszeiten&content=Wir%20haben%20Montag%20bis%20Freitag%20von%209%20bis%2017%20Uhr%20geoeffnet.&document_type=faq'; echo"
```

Expected: JSON with `"status":"indexed"` and `"chunk_count":1`.

- [ ] **Step 6: Confirm files saved** (no git repo for this project — see Global Constraints)

---

### Task 5: Wire up document search + end-to-end verification

**Files:**
- Modify: `Claude outputs/main.py` (`search_chunks`, `search_documents`)

**Interfaces:**
- Consumes: `get_embedding` (Task 3), `db_query` (existing).
- Produces: DB helper `search_chunks(query_embedding: list, limit: int) -> list[dict]`.

- [ ] **Step 1: Add the search DB helper**

In `Claude outputs/main.py`, add next to `insert_document_chunks`:

```python
async def search_chunks(query_embedding: list, limit: int):
    return await db_query(
        """
        SELECT dc.content, dc.chunk_index, d.id AS document_id, d.title,
               dc.embedding <=> %s::vector AS distance
        FROM document_chunks dc
        JOIN documents d ON d.id = dc.document_id
        ORDER BY distance ASC
        LIMIT %s
        """,
        (_vector_literal(query_embedding), limit),
        fetch="all",
    )
```

- [ ] **Step 2: Replace the `search_documents` endpoint**

In `Claude outputs/main.py`, replace the existing stub:

```python
@app.get("/api/v1/documents")
async def search_documents(query: str, limit: int = 10):
    """Search knowledge base using vector similarity"""
    if not pg_pool:
        raise HTTPException(status_code=503, detail="Database is not configured")
    if not query.strip():
        raise HTTPException(status_code=422, detail="query must not be empty")

    try:
        query_embedding = await get_embedding(query)
        rows = await search_chunks(query_embedding, limit)
        return {
            "query": query,
            "results": [
                {
                    "document_id": r["document_id"],
                    "title": r["title"],
                    "content": r["content"],
                    "chunk_index": r["chunk_index"],
                    "distance": float(r["distance"]),
                }
                for r in rows
            ],
            "count": len(rows),
        }
    except HTTPException:
        raise
    except Exception as e:
        logger.error(f"Document search error: {str(e)}")
        raise HTTPException(status_code=503, detail=f"Search failed: {str(e)}")
```

- [ ] **Step 3: Deploy**

```bash
SSHOPTS="-F /dev/null -o IdentitiesOnly=yes -i /c/Users/Jonny/.ssh/jarvis_core_key"
OUT="C:\Users\Jonny\Projekte\Claude\JARVIS\Claude outputs"
scp $SSHOPTS "$OUT/main.py" jarvis-core@72.61.186.98:/home/jarvis-core/jarvis/api/main.py
ssh $SSHOPTS jarvis-core@72.61.186.98 "cd /home/jarvis-core/jarvis && docker compose restart jarvis-api && sleep 8 && docker logs --tail 20 jarvis-api"
```

Expected: log ends with `Application startup complete.` (no traceback).

- [ ] **Step 4: End-to-end test (matches spec's manual verification requirement)**

Using the document uploaded in Task 4 Step 5 ("Wir haben Montag bis Freitag von 9 bis 17 Uhr geoeffnet."), search with a thematically related query:

```bash
ssh $SSHOPTS jarvis-core@72.61.186.98 "curl -s 'http://localhost:8000/api/v1/documents?query=Wann%20habt%20ihr%20auf%3F'; echo"
```

Expected: `results[0].title == "Oeffnungszeiten"` and `results[0].content` contains the opening-hours text, with the lowest `distance` of any returned result.

- [ ] **Step 5: Confirm files saved** (no git repo for this project — see Global Constraints)