Real-World Lessons from Shipping LLM-Based Software

Over the past year, we’ve been learning a lot about building and shipping LLM-based software, especially as our solutions have shifted toward agentic loops and tool calls. We’ve picked up hard lessons along the way around context management, evals, guardrails, and more.

That is why we are hosting an online mini-conference where four engineering leaders share the real-world lessons that came from building and running LLM-based software in production, at companies including Bridgewater Associates, Prudential Financial, and Morgan & Morgan.

This mini-conference covers:

Why context management is the bottleneck in production LLM systems, and strategies for addressing it.

Moving from retrieval-augmented generation (RAG) to page-by-page LLM extraction when processing massive documents like 5,000-page medical records, and from narrowly scoped LLM tasks toward more autonomous agents handling larger workflows.

Building and evaluating an AI agent platform at hedge-fund scale, avoiding over-engineered agent architectures and oversized eval suites, and where deterministic glue code beats agent-based orchestration.

Deploying a financial advisor AI assistant in a highly regulated industry: practical experience with context management, evals, judging, guardrails, and observability.

How to design permissions and security boundaries for agents and tools.

Practical evals, and how to avoid letting them get in the way of a good product.