One question, many moving parts
A user types a question into an AI-powered app, and a second later an answer appears. Behind that one interaction, the system may have:
- Authenticated the request and checked permissions
- Retrieved relevant documents and searched a vector database
- Loaded conversation history
- Called external tools or APIs
- Streamed generated tokens back to the user
- Enforced timeouts, rate limits and retries
If the app uses agents, it may also run code, read files, or chain several calls together before it responds.
Every step is a chance for latency to creep in, memory to balloon, or something to fail. With a few users, nobody notices. With thousands of concurrent requests, the infrastructure around the model decides whether the product feels fast and reliable or flaky and expensive. That is the layer where we increasingly reach for Rust.
The model is not the whole system
Most conversations about AI focus on models: which one, how big, how accurate. But a model is one component. In production it sits inside a system of gateways, queues, caches, retrieval pipelines, tool runners and observability. Teams often find their bottleneck isn’t the model at all. It’s the code around it.
Python is excellent for research, prototyping and model development. Its ecosystem is unmatched, and iteration speed matters when you’re experimenting. Infrastructure code has different priorities: steady latency, efficient memory use, safe concurrency, and behavior you can predict under load.
Why Rust fits infrastructure
Predictable performance. Rust has no garbage collector. Languages that rely on one can introduce pauses and extra memory overhead, which show up as latency spikes under load. Rust manages memory at compile time, so performance stays steadier.
Efficient resource use. A smaller CPU and memory footprint means more traffic on the same hardware, and at scale that becomes lower cloud bills. 37signals reported that parts of its HEY backend, rewritten in Rust with the help of AI agents, used roughly 99% less CPU and 95% less memory.
Memory safety. Memory-safety bugs are one of the most common sources of serious security vulnerabilities. Microsoft has reported that about two-thirds of the security bugs it tracked were memory-related. Rust’s compiler rules out whole categories of them.
Safe concurrency. AI workloads are highly concurrent: many requests, streams and tool calls in flight at once. Rust’s ownership model catches data races at compile time, so concurrent code is far less likely to fail in subtle ways.
Strong isolation. AI agents increasingly execute code they generated themselves. Rust compiles to WebAssembly, which lets you run that code in a sandbox with strict limits on memory, CPU and system access. If a tool misbehaves, the failure stays contained.
Where Rust shows up in an AI stack
You don’t need to rewrite everything. These are the places where Rust tends to pay off first:

- Inference gateways and routers: services in front of models that handle batching, load balancing, auth and streaming.
- Agent runtimes: long-running processes that manage state, call tools and enforce timeouts and budgets.
- MCP servers and tool layers: controlled, fast access to internal data and APIs for AI assistants.
- Data and retrieval pipelines: parsing, chunking and indexing, where throughput matters.
- Developer tooling: package managers, linters and coding agents, many now written in Rust.
- Edge and embedded deployments: environments with tight memory and no room for a heavy runtime.
The pattern that works is Python for the interface, Rust for the engine. Three short examples show what that looks like. They are simplified, so production code needs proper error handling and tests.
1. Bounded, time-limited tool calls
Agents often fire many tool calls at once. This caps concurrency and cancels slow calls, so one bad tool can’t stall everything.
use std::{sync::Arc, time::Duration};
use tokio::{sync::Semaphore, time::timeout};
async fn call_tool(name: &str) -> Result<String, String> {
// Call an API, database, or search index here
Ok(format!("{name}: done"))
}
async fn run_tool(limiter: Arc<Semaphore>, name: String) -> Result<String, String> {
// Wait for a slot: at most N tools run at once
let _permit = limiter.acquire().await.map_err(|e| e.to_string())?;
// Give up if the tool takes longer than 5 seconds
match timeout(Duration::from_secs(5), call_tool(&name)).await {
Ok(result) => result,
Err(_) => Err(format!("tool '{name}' timed out")),
}
}
2. A Rust hot path called from Python
Keep your research code in Python and move only the bottleneck. Here is a vector similarity function used in retrieval, exposed with PyO3.
use pyo3::exceptions::PyValueError;
use pyo3::prelude::*;
#[pyfunction]
fn cosine_similarity(a: Vec<f32>, b: Vec<f32>) -> PyResult<f32> {
if a.len() != b.len() {
return Err(PyValueError::new_err("vectors must be the same length"));
}
let dot: f32 = a.iter().zip(&b).map(|(x, y)| x * y).sum();
let norm_a = a.iter().map(|x| x * x).sum::<f32>().sqrt();
let norm_b = b.iter().map(|x| x * x).sum::<f32>().sqrt();
if norm_a == 0.0 || norm_b == 0.0 {
return Ok(0.0);
}
Ok(dot / (norm_a * norm_b))
}
#[pymodule]
fn fastvec(m: &Bound<'_, PyModule>) -> PyResult<()> {
m.add_function(wrap_pyfunction!(cosine_similarity, m)?)
}
Build it with maturin develop, then call it from Python:
import fastvec
score = fastvec.cosine_similarity([0.1, 0.9, 0.3], [0.2, 0.8, 0.4])
3. Sandboxing untrusted AI-generated code
Run generated code as WebAssembly with a fuel budget, so an infinite loop is stopped instead of hanging the host.
use wasmtime::*;
fn run_untrusted(wasm: &[u8]) -> Result<i32> {
let mut config = Config::new();
config.consume_fuel(true); // meter execution
let engine = Engine::new(&config)?;
let module = Module::new(&engine, wasm)?;
let mut store = Store::new(&engine, ());
store.set_fuel(10_000)?; // hard execution budget
// No imports: the guest can't touch files, network, or the host
let instance = Instance::new(&mut store, &module, &[])?;
let run = instance.get_typed_func::<(), i32>(&mut store, "run")?;
run.call(&mut store, ()) // traps if fuel runs out
}
If the code runs out of fuel or crashes, the failure stays inside the sandbox and the agent keeps running.
Why AI makes Rust easier to adopt
Rust’s biggest cost was never speed. It was developer productivity. The ownership and borrowing model has a steep learning curve, and rewrites in particular used to be slow, manual and expensive. AI coding tools shift that in two ways:
- They handle the mechanical translation. Converting straightforward code from one language to another is exactly the grunt work AI does well.
- The compiler acts as a built-in reviewer. In looser languages, AI-generated code can look correct while hiding runtime problems. In Rust, the compiler rejects many of those mistakes immediately: bad lifetimes, type mismatches and unsafe sharing fail at compile time, not in production.
The result is a useful loop. AI writes quickly, the compiler checks strictly, and the human engineer focuses on design and review.
What AI doesn’t fix
We’re enthusiastic about Rust, but careful about overselling it. Engineers who have done real migrations point to the same limits.
- Architecture still comes first. AI can translate code, but it can’t untangle a poorly structured system. If the design is wrong, Rust won’t save it.
- Verification becomes the bottleneck. Generation gets faster, but review doesn’t. AI can produce a mountain of code that someone still has to check against real system requirements.
- Interop weakens the guarantees. Real systems talk to C libraries and vendor SDKs. Wrapping them requires unsafe blocks, so Rust’s safety holds in your code and stops at that boundary.
- Not every vulnerability is a memory bug. Weak authentication, broken authorization and logic errors are just as dangerous in Rust.
- Determinism and compliance take work. A memory-safe task that misses its deadline has still failed. In regulated industries, certified Rust toolchains are younger than C’s, so a safer rewrite can still be harder to get through an audit.
- Toolchain support varies. Some embedded platforms don’t yet have full Rust support.
How we approach Rust adoption at Hacksprint
When teams ask whether they should move to Rust, we start with questions, not a rewrite plan:
- Where does it actually hurt? Profile first and find the component with the latency, cost or reliability problem.
- Is it a language problem or a design problem? The biggest wins often come from fixing architecture, caching or concurrency before touching the language.
- Can we isolate it? The best first candidates are well-bounded components with clear interfaces, like a gateway, a runtime or a tool server.
- Can we verify it? We set up tests, benchmarks and review practices before AI generates a single line, so we can measure whether the rewrite helped.
- Can the team maintain it? A rewrite only one person understands is a liability, so we plan knowledge transfer from day one.
Then we let AI draft the translation, lean on the compiler and tests to catch mistakes, and keep experienced engineers in charge of architecture and review.
A practical checklist
Before you commit to Rust for part of your AI stack:
- Identify the component with measurable latency, memory or cost problems
- Benchmark the current implementation so you have a baseline
- Choose a bounded component with a clean interface
- Decide how it will talk to existing Python or C code
- Plan for security beyond memory safety: authentication, authorization and input validation
- Budget time for human review of AI-generated code
- Confirm toolchain and compliance requirements for your target platforms
The bottom line
Rust isn’t replacing Python, and not every AI application needs a rewrite. Model training, experimentation and a great deal of application logic will stay where they are.
But as AI moves from demos to production, the infrastructure around the model matters more. For the performance-critical, security-sensitive and highly concurrent parts of that stack, Rust offers a combination that is hard to match: speed, safety and predictability. AI is lowering the cost of getting there.
The teams that benefit most won’t be the ones who rewrite the most code. They’ll be the ones who choose the right component, verify the results, and keep strong engineering judgment in the loop.
Planning an AI product or considering Rust for your stack? Talk to Hacksprint about an architecture review.
