Work SitimmVito search
Backend case · October 2026
Teaching an assistant to find the right answer.
SitimmVito is the assistant inside SITIMM's app. Over one weekend it went from a fixed folder of documents to searching everything the union publishes, inside the Postgres database the platform already had.
- My role
- Design, decisions and review. AI agents wrote the code.
- When
- One weekend, 24–25 October 2026
- Stack
- FastAPI, PostgreSQL full-text search, pgvector, Celery, Claude Haiku
- Result
- The right source for 13 of 16 test questions, up from 9
Screen
It started with an outage
On a Saturday the assistant's AI provider account ran out of credit. SitimmVito could still say hello and answer exact FAQ matches. Nothing else.
The first fix took a few hours: another model, the cheapest one that did the job, with keyword search over a fixed folder of documents. It worked.
It also showed the real problem. The assistant only knew what was in that folder, and the union publishes every day.
The decision: no new infrastructure
I had three options: go back to a hosted vector store, add a dedicated vector database, or use what was already running. I chose Postgres. It has full-text search and, with pgvector, vector search. For a team of one, one less service to run matters more than a slightly better ranking.
The cost of that choice: ranking quality became my problem.
- Cut into passagesNews, events, agreements, magazines, FAQs, laws and video transcripts.
- Index twiceA Spanish keyword index that ignores accents, and a vector.
- RefreshEvery 15 minutes, with the public API's filters. An event under embargo can't leak through the chat.
- Merge by rankKeyword and vector results together. If the vectors fail, keywords still answer.
Five ways it was wrong
| What I saw | Why | The fix |
|---|---|---|
| Common words beat the right answer | Asked about the Christmas bonus, magazine pages full of “days” and “December” outranked the labour-law article that answers it. Postgres's ranking has no idea which words are rare. | Each matching word scores by how rare it is in the index. The built-in rank only breaks ties. |
| That fix took 4 to 15 seconds | Postgres inlined a subquery and recounted the index for every matching row. | Two words: AS MATERIALIZED. 36 to 83 ms on a test database of 18,700 passages. |
| Filtered searches came back empty | A vector index takes its nearest matches first and filters afterwards. Asking only for agreements, 114 among about 7,000 passages, returned nothing. | pgvector's iterative scan keeps reading until the filter is satisfied. |
| Old news beat current facts | A dentist's offer from a 2016 magazine outranked the agreement in force. | Current sources get a boost when results are merged, and the prompt treats old magazine pages as history. |
| It refused without looking | A safety rule, “only answer SITIMM questions”, made it turn away questions about allied unions whose answers were in the index. | Search first, then decide what's off-topic. Unrelated requests and attempts to extract the prompt are still refused. |
How I knew
| Measure | Before | After |
|---|---|---|
| Test questions that found the expected source | 9 of 16 | 13 of 16 |
| Questions about allied organisations answered | 0 of 6 | 6 of 6 |
| Questions end to end through the public API | 17 of 18 | 18 of 18 |
| Keyword query, test database | 4–15 s | 36–83 ms |
Unit tests with fake models were green the whole time. One bug only showed up against the real API: on its last round the model wrote a tool call as plain text until it ran out of tokens. Only the evaluation against the real thing caught it.
What I'd do differently
- Write the evaluation first. I tuned ranking by feel for a few hours before I had 16 questions to measure against.
- Grow the evaluation. Sixteen questions catch the big failures, not slow drift. Three of them still miss.
- Plan for size. Word rarity is counted on every query. That holds to around 100,000 passages; after that it needs a statistics table.
Want the details?
I'm happy to walk through the queries, the evaluation and the parts that went wrong.