Development

Challenges in Production AI Systems

Challenges in Production AI Systems arise as AI moves from development to deployment, with issues including rapid code arrival, rising token costs, and the need for robust evaluations and secure agent management.

AI-generated editorial illustration of the article topic; not a photograph, product screenshot or measured result.

Challenges in Production AI Systems

Challenges in Production AI Systems arise as AI moves from development to deployment. According to InfoQ Homepage, once AI is in production, code arrives faster than it can be reviewed, token costs rise, and evaluations must be sufficiently robust to justify action. Additionally, agents require careful management of permissions and guardrails to ensure safe and effective operation. As noted in the QCon AI New York 2026 keynote, identity and authorization for agents in production present two problematic options for granting access. Furthermore, agent workloads that function well as prototypes can become prohibitively expensive in production due to increasing context requirements. These issues highlight the complexity of maintaining scalable, secure, and cost-effective AI systems in real-world environments.

Identity and Authorization for Agents

Identity and authorization for agents in production require careful consideration to avoid security risks. The QCon AI opening keynote highlights two problematic approaches to granting agents access, emphasizing the need for better strategies. Nancy Wang, founder and former leader of AWS Data Protection, outlines alternative solutions in her talk, When Software Becomes a User: Identity and Authorization for Agents in Production. She advocates for treating agents as users with defined roles and permissions, ensuring they operate within secure boundaries. Ronak Nathani, who led LinkedIn’s transition to Kubernetes, suggests implementing server-side guardrails, such as rate limits and peer approval for production changes, to add an extra layer of control. These approaches aim to balance flexibility with security, preventing unauthorized actions while enabling agents to perform their intended functions effectively.

Evaluations and Cost Management in Production

Evaluations and cost management are critical in production environments for agents and large language models (LLMs). According to Fabio Flores, who leads foundation development for Agentic Ordering at DoorDash, engineers should apply one judge to both simulated and production sessions to ensure that improvements measured before launch carry into production. This approach helps maintain consistency and reliability when deploying changes. In his talk, Make Each Token Count: Compression, Caching, and Retrieval for Production LLM Context, Flores discusses strategies such as tiered storage for conversation history and the failure modes his team encountered. These insights highlight the importance of managing LLM context efficiently to reduce costs and improve performance. Evaluations are useful when engineers trust the score enough to ship or hold on it, emphasizing the need for rigorous testing and validation before deployment.

Conference Structure and Attendee Engagement

The QCon AI New York 2026 conference, scheduled for December 15-16 at The Westin Jersey City Newport, is designed for senior engineers and technical leaders involved in production AI decision-making. The event structure emphasizes direct engagement between attendees and speakers, with a ratio of one speaker for every eleven attendees. This setup allows for in-depth discussions and provides time for follow-up questions from those who have implemented similar systems. The conference’s location, just five minutes by PATH train from the World Trade Center, ensures accessibility for participants. While the focus is on technical leadership and practical AI deployment, the event does not claim to cover all aspects of AI development or provide universal guarantees about attendee experiences. Specific details about session content or attendee satisfaction are not included in the provided evidence.

Learning Outcomes and Practical Takeaways

The Learning Outcomes and Practical Takeaways section of QCon AI New York 2026 emphasizes actionable insights for attendees. The QCon AI Program Committee specifically invites senior practitioners who are implementing strategies that other teams can learn from, ensuring that the content is grounded in real-world experience. The aim is for teams to return with patterns ready to adopt and costly mistakes they can avoid. All confirmed talks are available on the QCon AI New York 2026 sessions page, providing a resource for further exploration of the topics discussed.