The DNS Misconfiguration That Broke My Intelligent Infrastructure
A real-world diagnosis of how Docker networking, dangling env vars, and a misnamed container silently corrupted an arifOS vault
The Scene
It was past midnight. Three Docker containers — A-FORGE-arifos-mcp, A-FORGE-ollama, and qdrant — were all showing green. Healthy. Running. The Postgres vault was up for three hours without a restart.
And yet nothing was writing to memory. The audit ledger was empty. Every arifos_memory call was failing silently.
The system looked alive. It was, in fact, talking to no one.
The Setup: What arifOS Is
arifOS is a constitutional governance kernel for AI agents — a living law that sits between human intent and machine execution. It is not a chatbot wrapper. It is not a prompt engineering trick. It is an operating system.
At its core, arifOS runs a metabolic pipeline:
text
000 INIT → 111 SENSE → 333 MIND → 555 MEM → 666 HEART → 888 JUDGE → 999 SEAL
The SEAL stage writes a cryptographic verdict to a Merkle-chained vault ledger. If the vault cannot be reached, the audit trail breaks. And without an audit trail, the entire constitutional guarantee collapses — because every action is supposed to be inspectable, attributable, and logged.
That is what broke. And the reason was embarrassingly mundane.
The Symptoms
Three distinct failure signatures, all appearing at the same time:
1. Name or service not known on Ollama calls
The memory engine was trying to embed vectors using a local Ollama instance. Every call threw a DNS resolution error. But A-FORGE-ollama was healthy, port 11434 was bound. Something was wrong with how it was being addressed, not whether it was running.
2. Silent vault write failures
The vault’s PostgreSQL client was connecting — or so it appeared — but nothing was being persisted. The ledger read empty. No exception in the logs, just silence.
3. arifOS/arifosmcp/.env did not exist
The application’s local environment file had never been created from the example template. The house was built, but the water had never been turned on.
The Diagnosis: Three Interleaved Root Causes
Cause 1 — The Container Name vs. Service Name Mismatch
This is the subtle one. The docker-compose.yml defined the service as ollama. Docker Compose creates DNS resolution by service name — so internally, http://ollama:11434 should resolve. But the container was running with the name A-FORGE-ollama (due to a container_name: override in the Compose config).
When the MCP server tried to reach http://ollama:11434, the DNS lookup returned nothing. The service alias worked in Compose context, but something in the runtime resolution path was using the container name instead of the service name.
The fix applied: Append OLLAMA_URL=http://A-FORGE-ollama:11434 to the root .env so the explicit container hostname is used instead of the bare service alias.
The deeper lesson: container_name overrides in Docker Compose can silently shadow service-name DNS. When you explicitly name a container, alias resolution becomes ambiguous depending on what network mode and DNS resolver the other container is using. If an app reads OLLAMA_URL from env, it takes that value over the Compose-level injection — but only if that env file is actually loaded.
Cause 2 — The Missing .env That Was Never Created
The arifOS/arifosmcp/ directory had .env.example and .env.docker.example but no actual .env. The memory_engine.py calls os.getenv(“OLLAMA_URL”) with a default fallback. Without the env file, it used the hardcoded fallback: http://ollama:11434 — which, due to Cause 1, does not resolve.
More critically, DATABASE_URL in the example file was commented out:
text
# DATABASE_URL=postgresql://arifos_admin:***@postgres:5432/arifos_vault
Two problems in one line: it is commented out, and the password is wrong.
The fix applied: Copy .env.example to .env, uncomment DATABASE_URL, and replace the password with the correct encoded value.
The deeper lesson: Example files are promises. If a service relies on env vars that exist only in .env.example, a deployment is silently broken from day one unless someone actively materialises that file. This kind of failure does not error loudly — it fails softly, with silent defaults or silently wrong connections.
Cause 3 — The Vault DSN That Looked Right But Was Not
The root .env had DATABASE_URL pointing to postgres:5432. The postgres container was running healthy. But agents_66.py reads:
python
postgres_url=os.getenv(“ARIFOS_VAULT_URL”, os.getenv(“DATABASE_URL”))
Both env vars were present in the root .env. But the application-level .env — the one actually loaded by the MCP server process inside the container — had neither set correctly. The root .env was visible to the host, not to the container process.
The fix applied: Ensure the application-level .env has ARIFOS_VAULT_URL or DATABASE_URL set with the correct encoded password, not just the root .env on the host.
The deeper lesson: Host-level and container-level env resolution are completely separate. A variable in /root/.env is not automatically available inside a running container unless it is passed via env_file:, environment:, or mounted. Editing the host .env after docker compose up does nothing to the running container’s env state.
The Eureka Pattern
Taken individually, each of these failures is a known, solved problem. The insight comes from seeing them together: they form a compounding silence.
No single failure throws a loud exception. Each one degrades a specific channel:
DNS failure silences the embedding layer (no memory ingestion)
Missing .env silences the database connection (no vault write)
Wrong DSN password silences vault persistence (no audit trail)
Because arifOS is designed to degrade gracefully rather than crash — HOLD instead of VOID, silence instead of exception — a triple failure looks exactly like a system running perfectly while producing no durable output.
This is the governance risk of graceful degradation without telemetry gates. A system designed never to crash will sometimes silently not work.
The Audit Sequence (What to Run Before Any Ingest Test)
Before writing a single byte to a vault, verify the pipes from inside the container:
bash
# 1. Verify DNS resolution from inside the MCP container
docker exec A-FORGE-arifos-mcp getent hosts ollama qdrant postgres
# 2. Print active environment for critical vars
docker exec A-FORGE-arifos-mcp env | grep -E “OLLAMA|QDRANT|DATABASE|VAULT|SUPABASE”
# 3. Check for duplicate keys (appending can create conflicts)
grep -c “OLLAMA_URL” /root/.env
# 4. Verify the application .env was created and has the right vars
grep -E “URL|HOST|PORT” arifOS/arifosmcp/.env
# 5. Confirm Supabase key variable name matches what the code uses
grep “SUPABASE” arifOS/arifosmcp/agents_66.py | head -5
# 6. Confirm postgres is reachable by name from inside the container
docker exec A-FORGE-arifos-mcp pg_isready -h postgres -U arifos_admin
# 7. Restart the MCP container to pick up new env values
docker compose restart arifos-mcp
# 8. Tail logs and run the ingest smoke test
docker logs -f A-FORGE-arifos-mcp
Run these in order. Do not skip to step 8. The ingest call is the final seal, not the diagnostic tool.
The Principle Behind the Fix
arifOS operates under 13 Constitutional Floors. Floor F1 is Amanah — reversible first. Irreversible operations require an explicit 888 HOLD before execution.
The debug session violated F1 in a subtle way: it performed multiple environment file writes (appending to .env, running sed in place, copying example files) without first verifying the ground truth. Each edit was an irreversible state change to config files that a running container might be reading.
The correct protocol:
Read before writing. Verify what is actually loaded by the running process, not what exists on the filesystem.
One change, one verification. Make one config change, restart the relevant container, check the effect before proceeding.
888 HOLD on any write that cannot be trivially undone. Modifying a running system’s env without a backup of the original state is an irreversible operation.
Intelligence is not speed. The Eureka is not in the fix — it is in the sequence that makes the fix safe to apply.
What This Means for AI-Native Infrastructure
arifOS is built on a principle: every AI action must be inspectable, reversible where possible, and bounded by explicit rules before it touches the world.
That principle applies to the infrastructure itself. Container orchestration, environment injection, DNS resolution — these are not “just devops.” They are the substrate through which AI agents exercise agency. A misconfigured DNS entry is not a cosmetic bug. It is an agentic blindspot. The agent cannot reach its own tools, so it proceeds without them, silently, producing outputs with no grounding in external state.
The vault exists precisely to catch this: a Merkle-chained ledger that accumulates evidence of correct operation. When the vault is empty, the system has not failed loudly. It has failed constitutionally. No evidence means no accountability. No accountability means no trust.
Fix the DNS. Write to the vault. Then seal.
Arif Fazil is the Architect of arifOS — a sovereign, open-source, MCP-native governance kernel for AI agents. Built in Seri Kembangan, Malaysia.
Source code: github.com/ariffazil/arifOS
Live endpoint: arifosmcp.arif-fazil.com
DITEMPA BUKAN DIBERI — 999 SEAL ALIVE