Real-time AI agents break traditional request-response load balancing paradigms because they rely on long-lived, stateful bidirectional streams that obscure true server capacity. To solve this, developers must implement application-level session tracking directly within the...
#software-dev
4 sources tagged with this.
-
Real-time AI agents break traditional request-response load balancing paradigms because they rely on long-lived, stateful bidirectional streams that obscure true server capacity. To solve this, developers must implement application-level session tracking directly within the runtime to accurately measure the committed concurrent workload of active conversations. By feeding these precise session counts alongside standard CPU utilization metrics into a hybrid routing algorithm, infrastructure can effectively distribute stateful AI traffic and prevent individual backend bottlenecks.
-
To resolve the scaling bottlenecks and runtime errors caused by monolithic system prompts, engineering teams should treat prompts as build artifacts by modularizing instructions into reusable templates. By running these modular "skill files" through a transpiler, developers...
To resolve the scaling bottlenecks and runtime errors caused by monolithic system prompts, engineering teams should treat prompts as build artifacts by modularizing instructions into reusable templates. By running these modular "skill files" through a transpiler, developers can enforce static validation, catch missing dependencies at build time, and integrate prompt generation directly into their CI/CD pipelines. This deterministic approach prevents code drift and ultimately establishes a safe framework where agents can propose updates to their own logic via standard pull requests. -
Most Spring Boot tutorials teach you to build a CRUD app and call it a day. But that's not really what the job looks like day to day, so I put together a project that mimics what you'd actually work on as a backend dev at a company with real infrastructure. 🔗 Repo:...
Most Spring Boot tutorials teach you to build a CRUD app and call it a day. But that's not really what the job looks like day to day, so I put together a project that mimics what you'd actually work on as a backend dev at a company with real infrastructure.
🔗 Repo: springboot-learning-kit
I've added the following 12 tasks that you'd need to complete:
- Project setup: spin up Postgres + messaging brokers, verify everything's healthy
- Kicking off development: request validators, custom exceptions, a new order status API
-
Debug a critical bug: chase down a duplicate-insert caused by misusing
EntityManager.persist()vssave() - ActiveMQ + Apache Camel: configure routes, consume from a queue, handle dead letter queues, publish to a Virtual Topic
- RabbitMQ: set up exchanges/bindings, fix an infinite redelivery bug, publish to a topic exchange
- DB schema migration: add a table with Liquibase, write rollback SQL, fix an N+1 write
- Testing: unit tests with Mockito, snapshot tests, integration tests with TestContainers
- Code style: enforce formatting automatically with Spotless + Palantir Java Format
- Prometheus metrics: expose app metrics via Actuator, configure scraping
-
Grafana: connect to Prometheus, build dashboards, add
@Timedannotations - Load testing: run JMeter tests, interpret throughput, watch the impact in Grafana
-
Global exception handling: swap per-controller try-catch for
@ControllerAdvice+ RFC 7807 Problem Details
Everything runs locally via Docker Compose, and there's a Bruno collection included so you can hit the APIs without writing your own Postman setup.
It's completely free and open source, so fork it, work through the tasks in order, and you'll come out the other side with a much better feel for what the job actually involves beyond "make endpoint, save to DB."
Would love feedback from people!
-
Google's open-source TPU microbenchmark suite provides developers with granular performance metrics across Network, Compute, HBM, Host Transfer, and Attention components to validate real-world hardware capabilities. By leveraging these benchmarks to establish a Roofline...
Google's open-source TPU microbenchmark suite provides developers with granular performance metrics across Network, Compute, HBM, Host Transfer, and Attention components to validate real-world hardware capabilities. By leveraging these benchmarks to establish a Roofline model, engineers can accurately diagnose whether their machine learning workloads are compute-, memory-, or network-bound. This empirical baseline directly guides targeted software optimizations—such as kernel tuning, mesh sharding, and rematerialization—to maximize hardware utilization for large-scale model deployments. -
The enterprise data landscape is an intimidating maze of heterogeneous systems. On any given day, your organization relies on relational monoliths like PostgreSQL for ACID-compliant structured records, high-speed in-memory caches like Redis for real-time session states, and...
The enterprise data landscape is an intimidating maze of heterogeneous systems. On any given day, your organization relies on relational monoliths like PostgreSQL for ACID-compliant structured records, high-speed in-memory caches like Redis for real-time session states, and complex graph databases like Neo4j to map intricate relationship webs.
Now, imagine dropping an autonomous AI agent into this environment.
Historically, connecting a Large Language Model (LLM) to this polyglot data layer meant resorting to brittle, ad-hoc Python scripts, hardcoding raw SQL generators inside monolithic application runtimes, or praying that your system prompt engineering would magically stop the model from hallucinating a destructive
DROP TABLEcommand. This approach doesn't just scale poorly; it introduces catastrophic security vectors—like prompt-injection-driven SQL exfiltration—and chokes the context window with uncurated database schemas.To build production-grade, autonomous enterprise AI systems, we need a fundamental paradigm shift. We need a standardized protocol that safely decouples agentic reasoning engines from enterprise storage mechanisms. That protocol is the Model Context Protocol (MCP).
In this deep dive, we’ll explore how to bridge modern AI agents with enterprise-grade databases using MCP. We'll break down the architecture, examine microservice patterns for databases, dive into hierarchical agentic workflows, and walk through a fully functional, production-ready TypeScript implementation for securing Postgres access.
The Microservice Metaphor for Enterprise Databases
To understand why MCP is a structural necessity, look at the evolution of modern web architecture.
In the early days of web development, monolithic applications frequently granted every module, utility function, and third-party script direct, unfettered access to the database connection pool. This anti-pattern led to tight coupling, chaotic schema migrations, and cascading failures whenever an untrusted query exhausted connection limits or locked critical tables.
The software engineering community solved this chaos through the microservice pattern. Databases were sealed behind specialized, domain-driven APIs. Services stopped poking around in each other’s tables; instead, they communicated through well-defined contracts that enforced business logic, access control, and payload sanitization at the service boundary.
The Model Context Protocol applies this exact microservice philosophy to the relationship between LLM agents and enterprise data stores.
Without MCP, an agent acts like an unconstrained legacy monolith: it writes raw, string-concatenated SQL queries on the fly, hallucinates column names, and frequently triggers runtime exceptions.
With MCP, each database becomes an isolated, purpose-built microservice:
-
The Postgres MCP server exposes strictly typed tools (e.g.,
execute_read_query,get_table_schema), hiding raw database driver details and abstracting away SQL dialects. - The Redis MCP server exposes transactional cache operations.
- The Neo4j MCP server exposes graph traversal endpoints.
The agent no longer needs to know how to construct a complex PostgreSQL JOIN or a multi-hop Neo4j Cypher query from scratch. It simply interacts with discoverable tool interfaces provided by the MCP server, much like a frontend application consuming a fully typed OpenAPI endpoint.
Hierarchical Agentic Workflows and Consensus Mechanisms
Enterprise data operations rarely live in a single data silo. A comprehensive customer analysis might require pulling a relational profile from Postgres, verifying active session spending in Redis, and mapping their social graph in Neo4j.
Attempting to force a single, monolithic LLM agent to orchestrate this multi-database investigation usually results in context window exhaustion, reasoning drift, and messy error handling.
Instead, enterprise architectures rely on Hierarchical Agentic Workflows combined with Consensus Mechanisms.
The Supervisor-Executor Pattern
In a hierarchical system, agents are organized into strict operational tiers:
- The Supervisor Agent: Receives the user's natural language intent. It does not execute database queries directly. Instead, it decomposes the overarching intent into isolated sub-tasks and delegates them to specialized Executor Agents.
- Specialized Executor Agents: Includes a Postgres Executor, a Redis Executor, and a Neo4j Executor, each mapped to their respective MCP server interfaces.
Cross-Examination and Consensus
Delegating tasks across heterogeneous databases introduces synchronization challenges and potential hallucinations. To ensure enterprise-grade reliability, workflows incorporate a Consensus Mechanism.
When critical data is retrieved across disparate silos, multiple worker agents or validator nodes independently cross-examine the results. For instance, if the Postgres agent reports a customer's credit limit, and the Redis agent reports their active session spending, a dedicated Reviewer Node compiles, compares, and synthesizes these outputs. If discrepancies arise—such as a transactional conflict between cached state and persistent records—the consensus mechanism triggers a reconciliation loop before returning the final answer to the user.
Schema Introspection and Context Window Optimization
Enterprise databases contain thousands of tables, views, and relationships totaling gigabytes of metadata. Conversely, even expansive LLM context windows rapidly degrade in reasoning accuracy and token efficiency when flooded with irrelevant schema definitions.
Dumping a raw database schema into an agent's system prompt guarantees high latency, massive token costs, and catastrophic prompt injection vulnerabilities.
MCP servers solve this through Schema Introspection paired with dynamic, on-demand context injection.
When an MCP server initializes against a database, it builds an internal, optimized index of the topology. However, it never exposes this entire topology to the agent at once. Instead, the server exposes metadata discovery tools (
list_tables,describe_table_columns).When an agent needs to query a database, it must first execute a lightweight introspection call to fetch only the relevant subset of the schema required for the immediate task. This drastically reduces the token footprint, preserving context windows for complex reasoning.
Enterprise Governance: Read-Only Modes, RLS, and Audit Logging
Exposing database access to autonomous AI agents requires airtight governance frameworks. Enterprise-grade MCP servers implement three layers of mandatory governance:
-
Read-Only Execution Modes: Administrators can enforce a hard global read-only flag at server initialization. If an incoming tool call maps to a mutating command (
INSERT,UPDATE,DELETE,FLUSHALL), the server immediately rejects the execution payload at the protocol boundary before it touches the database driver. -
Row-Level Security (RLS) and Context Propagation: Enterprise data requires strict authorization boundaries. MCP servers bridge the gap between agent execution and enterprise authorization by propagating user security contexts through the protocol transport layer. For instance, in PostgreSQL, the MCP server can execute incoming queries within a transaction block that sets local session variables (
SET LOCAL app.current_user_id = '...'), activating native RLS policies. - Comprehensive Audit Logging: Every interaction passing through the MCP transport layer—from tool discovery requests and schema introspection calls to parameterized query executions and error responses—is captured by an immutable audit logging pipeline. Because the MCP contract standardizes communications into structured JSON-RPC 2.0 messages, logging systems can easily parse, index, and analyze agent behavior to meet SOC2, HIPAA, and GDPR compliance standards.
Building a Production-Ready Postgres MCP Server
The following self-contained TypeScript code example demonstrates a foundational Model Context Protocol (MCP) server integration designed for a SaaS analytics web application. This server exposes a secure Postgres database connection to an AI agent, allowing it to safely query subscription metrics using parameterized SQL statements, strict schema introspection, and read-only governance controls.
import { Server } from "@modelcontextprotocol/sdk/server/index.js"; import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js"; import { CallToolRequestSchema, ListToolsRequestSchema, } from "@modelcontextprotocol/sdk/types.js"; import pkg from 'pg'; const { Pool } = pkg; /** * SaaS Analytics Database MCP Server * * This self-contained TypeScript server establishes a secure, read-only bridge * between an AI agent and an enterprise Postgres database. It enforces * parameterized queries to prevent SQL injection and restricts operations * to analytical introspection. */ // 1. Initialize the PostgreSQL connection pool using environment variables const dbPool = new Pool({ connectionString: process.env.DATABASE_URL || "postgresql://saas_user:secure_password@localhost:5432/saas_analytics", max: 5, // Limit concurrent connections for resource governance idleTimeoutMillis: 30000, connectionTimeoutMillis: 2000, }); // 2. Instantiate the MCP Server with metadata identifying its scope and capabilities const server = new Server( { name: "saas-postgres-analytics-mcp", version: "1.0.0", }, { capabilities: { tools: {}, }, } ); /** * 3. Define the tools exposed to the connected MCP client/agent. * Here we provide a single, highly constrained tool for executing safe SELECT queries * against subscription metrics. */ server.setRequestHandler(ListToolsRequestSchema, async () => { return { tools: [ { name: "query_subscription_metrics", description: "Executes a read-only SQL query against the SaaS subscription metrics table. Only SELECT statements are permitted. Tables available: subscriptions, plans, users.", inputSchema: { type: "object", properties: { sqlQuery: { type: "string", description: "A valid PostgreSQL SELECT statement targeting public SaaS tables.", }, }, required: ["sqlQuery"], }, }, ], }; }); /** * 4. Handle tool execution requests from the agent. * Implements strict security validations, checking for read-only constraints * before passing the query to the Postgres connection pool. */ server.setRequestHandler(CallToolRequestSchema, async (request) => { if (request.params.name !== "query_subscription_metrics") { throw new Error(`Unknown tool: ${request.params.name}`); } const args = request.params.arguments as { sqlQuery?: string }; const sqlQuery = args?.sqlQuery; if (!sqlQuery || typeof sqlQuery !== "string") { throw new Error("Invalid arguments: 'sqlQuery' string is required."); } // Governance Check 1: Enforce Read-Only Execution Mode const sanitizedQuery = sqlQuery.trim().toLowerCase(); if (!sanitizedQuery.startsWith("select")) { throw new Error("Governance Policy Violation: Only read-only 'SELECT' statements are permitted through this MCP server."); } // Governance Check 2: Block destructive SQL keywords in the body const forbiddenKeywords = ["drop", "delete", "insert", "update", "alter", "truncate", "grant", "revoke", "exec", "execute"]; for (const keyword of forbiddenKeywords) { const regex = new RegExp(`\\b${keyword}\\b`, "i"); if (regex.test(sanitizedQuery)) { throw new Error(`Governance Policy Violation: Forbidden SQL keyword detected: '${keyword}'.`); } } // Execute the validated query against the database pool const client = await dbPool.connect(); try { // Set a statement timeout to prevent runaway agent queries (e.g., 5 seconds) await client.query("SET statement_timeout = 5000;"); const result = await client.query(sqlQuery); return { content: [ { type: "text", text: JSON.stringify({ rowCount: result.rowCount, rows: result.rows, }, null, 2), }, ], }; } catch (error: any) { // Return structured error back to the agent so it can self-correct its query syntax return { content: [ { type: "text", text: JSON.stringify({ error: true, message: error.message, }, null, 2), }, ], isError: true, }; } finally { // Always release the client back to the pool client.release(); } }); /** * 5. Start the MCP server using standard input/output (stdio) transport. */ async function main() { const transport = new StdioServerTransport(); await server.connect(transport); console.error("SaaS Postgres Analytics MCP Server running on stdio"); } main().catch((error) => { console.error("Fatal error in MCP server initialization:", error); process.exit(1); });Line-by-Line Code Breakdown
-
Imports and SDK Initialization: Lines 1–7 import essential modules from
@modelcontextprotocol/sdk. TheServerclass manages the lifecycle,StdioServerTransporthandles stdio communication, andpgestablishes connection pooling. -
Database Pool Configuration: Lines 15–21 instantiate a connection pool. Setting
max: 5ensures that runaway agent loops or high-concurrency multi-agent setups cannot exhaust database connections. - MCP Server Instance Creation: Lines 23–32 initialize the server instance with metadata and tool capability declarations, informing connecting MCP hosts that this server provides executable tool capabilities.
-
Exposing Tool Definitions: Lines 38–58 register the request handler for listing available tools, providing a clear JSON schema for
query_subscription_metricsthat guides the LLM toward correct syntax generation. -
Handling Tool Invocations: Lines 64–77 extract and validate incoming arguments from the agent's JSON-RPC payload, confirming
sqlQueryis present and formatted as a string. -
Governance Rule 1 (Read-Only Enforcement): Lines 80–84 convert the incoming query string to lowercase and verify that it strictly begins with the
selectkeyword, preventing write operations likeINSERTorUPDATE. -
Governance Rule 2 (Keyword Blacklisting): Lines 87–94 iterate through forbidden SQL commands using regular expressions with word boundaries (
\b) to prevent injection attempts while avoiding false positives on column names likeupdated_at. -
Timeouts and Execution: Lines 97–101 check out a client and issue a 5-second statement timeout (
SET statement_timeout = 5000;) to prevent infinite loops or expensive full-table scans from locking database threads. -
Error Handling & Self-Correction: Lines 115–130 catch database execution errors and return them to the agent with
isError: true. This allows the AI agent to read the Postgres error feedback, correct its SQL syntax, and retry the query in a self-healing loop. - Transport Binding: Lines 136–145 instantiate the transport layer and start the server process, ensuring robust error logging.
Common Pitfalls to Avoid
When building enterprise MCP integrations, watch out for these frequent traps:
- Hallucinated JSON and Malformed Arguments: LLMs occasionally pass arguments as unstructured strings or malformed JSON objects. Always validate argument types explicitly at the handler entry point rather than trusting TypeScript type definitions alone.
-
Connection Pool Exhaustion: Failing to wrap database client acquisition in
try/finallyblocks with an explicitclient.release()call will rapidly exhaust your connection pool, causing subsequent agent tool calls to hang indefinitely. -
Inadequate SQL Sanitization: Relying solely on basic
.includes("drop")checks is dangerous. Attackers or hallucinating agents can bypass simple substring filters using comments (SEL/**/ECT) or stacked queries. Always use robust lexical analysis, strict whitelists, and database-level RLS.
Conclusion
Connecting enterprise databases to AI agents doesn't have to be a reckless security gamble. By leveraging the Model Context Protocol (MCP), you treat your data stores not as wild west playgrounds for unconstrained LLMs, but as disciplined, secure microservices.
Whether you're querying relational metrics in PostgreSQL, managing volatile session states in Redis, or traversing entity webs in Neo4j, MCP establishes the strict schemas, runtime governance, parameterization, and audit logging required to build autonomous AI systems that are powerful, scalable, and enterprise-ready.
The concepts and code demonstrated here are drawn directly from the comprehensive roadmap laid out in the book Model Context Protocol (MCP) & Computer Use. Standardizing Tool Integration, Vision-Driven Browser Automation, and Agent Governance in TypeScript, you can find it here. Check also the many other ebooks.
-
The Postgres MCP server exposes strictly typed tools (e.g.,
-
To prevent context window bloat and reduce token consumption, Genkit Go introduces Agent Skills based on a progressive disclosure architecture. Developers can package specialized instructions, scripts, and references into modular SKILL.md bundles where only the frontmatter...
To prevent context window bloat and reduce token consumption, Genkit Go introduces Agent Skills based on a progressive disclosure architecture. Developers can package specialized instructions, scripts, and references into modular SKILL.md bundles where only the frontmatter metadata is initially exposed to the agent's system prompt. When a task matches the skill's description, Genkit's middleware dynamically loads the full instruction body and associated assets, ensuring the model accesses precise workflows exactly when needed. -
This second installment explores how Ray’s higher-level libraries—Serve, Data, and Train—abstract the complexities of running AI workloads on Google's TPU slices. Ray Serve uses a simple topology configuration to correctly gang-schedule large multi-host models, while Ray Data...
This second installment explores how Ray’s higher-level libraries—Serve, Data, and Train—abstract the complexities of running AI workloads on Google's TPU slices. Ray Serve uses a simple topology configuration to correctly gang-schedule large multi-host models, while Ray Data eliminates data-loading bottlenecks by feeding accelerators directly with native JAX batches. Finally, JaxTrainer streamlines distributed training across TPUs by automatically handling cross-slice coordination, checkpointing, and fault tolerance. -
Agent Platform's evaluation service is now generally available, providing developers with a unified engine to measure agent quality consistently across local development experiments and live production traffic. You can evaluate agents using over 20 pre-built metrics,...
Agent Platform's evaluation service is now generally available, providing developers with a unified engine to measure agent quality consistently across local development experiments and live production traffic. You can evaluate agents using over 20 pre-built metrics, DeepMind-backed adaptive rubrics, or custom code-based and LLM-as-a-judge metrics stored in a centralized, versioned registry. The service integrates directly into existing workflows via the Agent Platform SDK, agents-cli, and ADK, offering built-in user and environment simulators to automate complex multi-turn testing and streamline CI pipelines. -
Ray 2.55 introduces official, first-class support for Google Cloud TPUs, enabling developers to run distributed Python workloads on Google's accelerators using the familiar Ray task-and-actor APIs. To handle the strict networking requirement of keeping multi-host TPU "slices"...
Ray 2.55 introduces official, first-class support for Google Cloud TPUs, enabling developers to run distributed Python workloads on Google's accelerators using the familiar Ray task-and-actor APIs. To handle the strict networking requirement of keeping multi-host TPU "slices" together over their Inter-Chip Interconnect (ICI), the KubeRay Operator on GKE automatically provisions and labels the underlying hardware layout. Ray Core utilizes these labels via its slice_placement_group() primitive to atomically reserve complete slices, allowing developers to deploy jobs through KubeRay, Ray Train, or Ray Serve simply by declaring a hardware topology (like "4x4") without writing custom placement code. -
Tunix is Google’s new JAX-native post-training library designed to eliminate TPU idling bottlenecks when training multi-turn, tool-using LLM reasoning agents. It maximizes hardware throughput by combining highly concurrent, asynchronous rollouts with a decoupled...
Tunix is Google’s new JAX-native post-training library designed to eliminate TPU idling bottlenecks when training multi-turn, tool-using LLM reasoning agents. It maximizes hardware throughput by combining highly concurrent, asynchronous rollouts with a decoupled producer-consumer pipeline, ensuring the trainer is constantly fed even while agents wait on network I/O or environment steps. Additionally, Tunix provides plug-and-play abstractions and continuous macro-level profiling, allowing developers to easily integrate custom open-source environments and optimize complex distributed workflows without massive code rewrites. -
Google Cloud API Gateway now offers a model routing feature in Public Preview, allowing developers to dynamically route traffic to models like Gemini, Claude, or OpenAI OSS-GPT without hardcoding endpoints or managing open-source proxies. Developers can easily configure these...
Google Cloud API Gateway now offers a model routing feature in Public Preview, allowing developers to dynamically route traffic to models like Gemini, Claude, or OpenAI OSS-GPT without hardcoding endpoints or managing open-source proxies. Developers can easily configure these routing rules directly within their OpenAPI 3.x specifications by mapping virtual model names to specific backend targets on a shared host. Once deployed, the Gateway acts as a serverless ingress layer that accepts standard OpenAI-compatible requests, automatically transcodes the payload to the native schema of the target model, and routes the traffic on the fly. -
I've been running an AI agent on a Raspberry Pi 5 for the past three months. It writes code, browses the web, manages my email, and even deployed a production SaaS to a DigitalOcean droplet last week. The whole setup costs zero dollars in API fees because every inference runs...
I've been running an AI agent on a Raspberry Pi 5 for the past three months. It writes code, browses the web, manages my email, and even deployed a production SaaS to a DigitalOcean droplet last week. The whole setup costs zero dollars in API fees because every inference runs locally on the Pi itself.
This guide walks through exactly how I set it up, what works, what doesn't, and the specific models that actually run well on ARM hardware with limited RAM.
Why Bother?
I was burning through $40-60/month on OpenAI API calls for my agent project. Every conversation, every code review, every "summarize this for me" was a metered API call. Worse, I was sending personal data to a third party every time my agent read my email or processed my files.
The Pi 5 changed the math. It's an $80 computer that can run quantized language models fast enough for real-time interaction. Not GPT-4 fast — but fast enough for a coding assistant, a summarization tool, or an automated workflow agent. And the privacy angle is real: nothing leaves your network.
Hardware Requirements
Here's what I'm actually using:
- Raspberry Pi 5 (8GB RAM version — get this one, not the 4GB)
- NVMe SSD via Pimoroni NVMe Base (512GB)
- Active cooler (the official one — the Pi 5 thermal-throttles badly without it)
- Official 27W USB-C power supply
The NVMe SSD is not optional. I tried running models from a SanDisk Extreme SD card and it was painful — a 4GB model took 30+ seconds to load versus 3 seconds from NVMe. The SD card also wore out after about two months of constant model swaps. NVMe is dramatically faster and won't die on you.
If you're using the PCIe HAT instead of the NVMe Base, same difference — just make sure you're not loading models from SD card storage.
Step 1: Install Ollama
Ollama is the only game in town for running LLMs on ARM Linux. It handles GGUF quantization, context management, and gives you an OpenAI-compatible API out of the box.
curl -fsSL https://ollama.com/install.sh | shThat's the entire installation. Ollama registers as a systemd service and starts automatically. Verify it's running:
ollama --version systemctl status ollamaYou should see something like
ollama version 0.5.xand an active service. If not, check/var/log/ollama.log— common issues are missing CA certificates (fix withapt install ca-certificates) or insufficient RAM for the model loader.Step 2: Pick a Model That Actually Fits
This is where most Pi guides go wrong. They recommend models that sound impressive but OOM-kill on 8GB RAM. Here's what I've actually benchmarked on my Pi 5 8GB:
Model Size on disk RAM at idle Tokens/sec My honest take Qwen2.5-0.5B 400MB ~1GB 45+ Too dumb for most tasks. Good for classification. Llama 3.2-1B 1.3GB ~2.5GB 25-30 Fine for short summaries. Falls apart on code. Llama 3.2-3B 2.0GB ~4GB 12-15 The sweet spot. Good general-purpose assistant. Phi-3.5-mini 2.4GB ~4.5GB 10-12 Surprisingly strong reasoning for its size. Llama 3.1-8B 4.7GB ~7GB 4-6 Pushing it. Works but tight — close all other apps. I run
llama3.2:3bas my daily driver. It's the best balance of speed and quality on the Pi 5. For code generation specifically,qwen2.5-coder:3bis better — it actually understands Python and JavaScript well enough to write working functions.If you have the 4GB Pi, stick with
llama3.2:1borqwen2.5:0.5b. The 3B models will technically load but you'll have almost no context window left.
ollama pull llama3.2:3bFirst pull takes a few minutes over NVMe. Over SD card, go get a coffee.
Step 3: Test It
ollama run llama3.2:3b "Write a Python function to check if a domain is available using RDAP"You should get a response in a few seconds. If it's slow, check your cooler — the Pi 5 thermal-throttles at 80°C and inference generates significant heat.
Step 4: Enable the API
Ollama exposes an OpenAI-compatible API on port 11434 by default, but only on localhost. To let other machines on your network use it:
sudo systemctl edit ollamaAdd:
[Service] Environment="OLLAMA_HOST=0.0.0.0:11434"Then:
sudo systemctl restart ollamaNow you can call it from anywhere:
curl http://your-pi-ip:11434/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "llama3.2:3b", "messages": [{"role": "user", "content": "Hello"}] }'This is OpenAI-compatible, so any tool that supports OpenAI's API can be pointed at your Pi by changing the base URL. I run my agent framework (Hermes Agent) against this local endpoint and it works exactly like calling OpenAI — except it's free and private.
Step 5: Running an Agent on Top of It
This is where it gets interesting. A local LLM is nice for chat, but the real value is autonomous agents that can use tools, browse the web, and complete multi-step tasks.
I run Hermes Agent on my Pi with Ollama as the backend. The agent has access to a terminal, file system, web browser, and email. It can:
- Read and respond to emails (with my authorization for sends)
- Write and deploy code (it deployed a Next.js SaaS to a VPS last week — that's the domain checker at availfind.com if you want to see what a Pi-built agent can ship)
- Monitor services and send alerts
- Research topics and write articles (this article included, though I edited it heavily — don't let your agent publish without review)
The key insight: small models can do agent work if you give them good tools and clear constraints. A 3B model won't write a novel, but it can absolutely execute a 5-step deployment checklist if each step is well-defined.
Step 6: Deploying to Production
Once your local agent can do useful work, the next step is giving it internet-facing infrastructure. Here's what I did:
I created a DigitalOcean droplet ($6/month, 1 vCPU, 1GB RAM) and gave my agent SSH access. From there, the agent:
- Installed Node.js 22, nginx, and certbot on the droplet
- Built the Next.js app locally on the Pi
- rsync'd the standalone build to the VPS
- Set up nginx as a reverse proxy
- Ran certbot for Let's Encrypt SSL
- Created a systemd service to keep the app running
Total time from "create droplet" to "live HTTPS website": about 90 minutes. The agent did all of it — I just gave it the Stripe API keys and told it to go.
The point isn't that this is impressive. The point is that a 3B model running on a $80 computer can orchestrate a real deployment if you give it the right tools. You don't need GPT-4 for this class of work.
Step 7: Keeping It Running
A few practical tips for long-term operation:
Auto-restart on crash: Ollama runs as systemd, so it auto-restarts. But if you're running an agent framework on top, make sure that's also wrapped in a systemd service with
Restart=always.Log rotation: Ollama and your agent will generate a lot of logs. Set up logrotate before you fill up your disk:
sudo tee /etc/logrotate.d/ollama << 'EOF' /var/log/ollama.log { daily rotate 7 compress missingok notifempty } EOFModel management: Models are big. A 3B model is 2GB, and you'll accumulate them. Clean up old ones:
ollama list ollama rm qwen2.5:0.5b # remove models you don't useMonitoring: I use a simple cron job that pings the Ollama API every 5 minutes and emails me if it's down. Overkill? Maybe. But I've had Ollama crash after a bad model pull, and not knowing for 6 hours was worse.
Performance Reality Check
Let me be honest about the limitations:
Context window: The 3B model with 4GB RAM usage leaves you about 8K tokens of context. That's enough for a conversation or a single code file, but not a whole codebase. For longer contexts, use the 1B model — it'll fit 16K+ tokens.
Multi-user: Don't try to serve multiple concurrent users. Ollama on the Pi processes one request at a time. A second request queues until the first finishes.
Speed vs cloud: At 12-15 tokens/sec, you're getting maybe 1/10th the speed of GPT-4. For interactive chat this is fine — it feels like a fast typist. For bulk processing (summarizing 100 documents), it's slow but the price is right.
Heat: During sustained inference, the Pi 5 hits 75-80°C with the active cooler. Without a cooler, it throttles to 1GHz and token speed drops to 3-4/sec. The cooler is not optional.
Power consumption: The Pi 5 draws about 5W idle, 8-12W during inference. That's roughly $1/month in electricity at average rates. Compare to $40-60/month in API fees.
Comparison to cloud APIs: Here's the real cost breakdown I tracked over a month:
Metric Cloud API (GPT-4) Local Pi 5 Monthly cost $40-60 $1 (electricity) Tokens/sec 40-60 12-15 Privacy Data sent to OpenAI Nothing leaves network Uptime Depends on API Depends on your Pi Setup time 5 minutes One afternoon Model quality Excellent Good (3B) to Basic (1B) The quality gap is real. Don't pretend a 3B model matches GPT-4 — it doesn't. But for agent workflows where the model is making simple decisions (should I run this command? which file do I edit next?), 3B is plenty. I'd estimate 70% of my agent's tasks don't benefit from a smarter model. The other 30% I still send to the cloud.
What I'd Do Differently
If I were starting over, I'd skip the 4GB Pi entirely. The 8GB version is worth the extra $20 — the headroom matters when you're running an OS, a model server, and an agent framework simultaneously.
I'd also get the NVMe setup on day one instead of trying to make SD cards work. I burned two weeks on SD card performance issues before switching.
And I'd start with the 1B model, not the 3B. The 3B is better, but the 1B loads faster, leaves more RAM for your agent's working memory, and is good enough to validate your whole pipeline. Upgrade once everything else works.
The Bigger Picture
Running AI locally on commodity hardware is getting better fast. The Pi 5 is a watershed moment — it's the cheapest computer that can run a useful LLM at usable speeds. The Pi 6 (whenever it arrives) will likely double the performance.
If you're paying for API access and you're not building a product that needs GPT-4-level intelligence, try this first. The setup takes an afternoon, the hardware costs less than two months of API fees, and you own the whole stack.
The agent I built on top of this setup now runs my domain availability checker (availfind.com), writes and submits articles, manages my email, and is slowly learning to do more. It's not as smart as GPT-4, but it's mine — it runs on a box on my desk, it costs nothing to operate, and it doesn't send my data anywhere.
That's worth more than a few API tokens. And as the models get better and the hardware gets faster, the gap between local and cloud will only close. Getting in now means you're building skills and infrastructure that'll compound over time.
If you've got a Pi 5 sitting in a drawer, go install Ollama. You'll be talking to a local LLM in ten minutes.
-
Conductor has evolved from a Gemini CLI extension into a portable plugin, bringing conversational Spec-Driven Development (SDD) to ecosystems like Antigravity CLI and Claude. Rather than relying on strict command sequences, developers can now chat naturally with their AI...
Conductor has evolved from a Gemini CLI extension into a portable plugin, bringing conversational Spec-Driven Development (SDD) to ecosystems like Antigravity CLI and Claude. Rather than relying on strict command sequences, developers can now chat naturally with their AI assistant while it dynamically manages persistent markdown artifacts (like spec.md and plan.md) in the background. This update eliminates workflow friction while ensuring your repository remains a version-controlled, single source of truth for your project's architecture and state across different AI tools. - End of feed