Shared AI Platform Backend

The challenge

Four products needed clinical-note generation, grounded chat, page design, agentic automation and embedding pipelines. Building that four times would have meant four bills, four sets of prompts, and four places for a tenant's data to leak.

What we built· with the Bracket Mind team

• Sandboxed code execution: the model writes Python or Node, and it runs in a fresh Docker container — 1 CPU, 512 MB, 256 processes, non-root, no credentials inside — force-removed after the run, with the agent's steps streamed to the browser over SSE
• Model routing from the database across eight provider types — Bedrock, Azure OpenAI, Anthropic, OpenAI, Gemini, NVIDIA NIM, Hugging Face and any OpenAI-compatible endpoint — failing over to the next ranked model on transient errors
• Agents tenants extend without a deploy: markdown skills loaded on demand, HTTP tools defined as data, and MCP servers whose tools are discovered at the start of each run
• Human approval for risky tool calls — write requests and every MCP tool pause the run and ask a person over the same SSE stream; a denial or a five-minute timeout tells the model no
• User-defined tools locked down against SSRF: hostnames resolved and private, loopback and cloud-metadata addresses refused, redirects never followed, responses capped, credentials encrypted at rest
• A RAG pipeline — embeddings, Pinecone retrieval, Cohere reranking on Bedrock, then synthesis — behind chat-abuse guardrails covering prompt injection, toxicity, credential probing, IP reputation and persistent bans
• One AI gateway for all four products: providers, models and per-task model choices set in an admin console without a deploy; every paid call logged with tokens, latency and cost, estimates flagged as estimates; cost reports by product, user, model and task; an operations dashboard for AI spend, errors, job queues and abuse blocks; and a daily check of model rates against published price lists
• Per-call AI cost attributed to the right tenant, and deny-by-default access to every AI feature, granted per tenant

How it fits together

Which engineering decisions shaped it?

1. 01A capability map, not Microsoft.Extensions.AI, decides what each provider can do

   Instead of

   Microsoft.Extensions.AI's common chat interface as the one contract for every provider

   Why

   That common surface can't carry prompt caching, extended thinking, streamed tool events or response prefill, which the cache-heavy and agentic features depend on; today only the Bedrock path carries them. Microsoft.Extensions.AI still runs inside every other provider, and the map decides in one place what each one can take, logging any feature it drops instead of losing it silently.

2. 02Every MCP tool call waits for a person

   Instead of

   Gating only calls that write, as we do for HTTP tools, or trusting an MCP server once it is registered

   Why

   An MCP server's tool names, descriptions and results are written remotely and can change after registration, so we treat them as untrusted. A denial or a five-minute timeout reaches the model as a refusal it plans around, and a run with no person on the stream gets no MCP tools at all.

3. 03MCP results are capped at 24,000 characters

   Instead of

   Passing each result through whole and trusting servers to keep them small

   Why

   An agent loop re-sends every earlier tool result on each round, so one oversized payload is billed again and again: a single 654K-character result cost 2.1M input tokens, about 28 times a normal page-builder run. A capped result is rebuilt to stay valid JSON, with a note telling the model what was cut.

4. 04The code sandbox keeps its network on and holds no credentials

   Instead of

   A sandbox with networking switched off

   Why

   The agent writes code to fetch and scrape web pages, which a network-off container can't do. So the boundary is what's inside: no API keys or tokens, and anything that needs a credential runs as a tool in the API process. Each run gets a fresh container that is force-removed afterwards.

5. 05Agent runs stop at 200 tool calls, a limit we set ourselves

   Instead of

   The agent framework's default cap of 40 iterations (model round trips) per request

   Why

   A migration makes one create-project call per project, so 40 couldn't finish a 67-project job, and when the framework's cap trips, its closing call is rejected by Bedrock. Our 200-call limit is a cost circuit-breaker for runaway retry loops, and the framework's own cap sits just above it, so ours always fires first.

Our founder's part

Haseeb Attaullah: Backend engineer on the AI layer — the sandboxed code-execution tool, the provider registry and its failover policy, tenant-defined agent skills, HTTP tools and MCP server connections, human approval for risky tool calls, deny-by-default AI task access, cost attribution for agents and background jobs, and encrypted per-tenant API keys. The gateway's routing service, cost reports and dashboard were built by a teammate.