Evaluating AI agents for NL-to-SQL generation across Azure Databricks AI/BI Genie, GitHub Copilot CLI, and Microsoft Agent Framework. We achieved ~75% accuracy with schema documentation and runtime validation, while discovering that business logic errors represent a fundamental limitation requiring domain expertise.

This blog post discusses the challenges and solutions encountered by the ISE team at Microsoft while making a distributed system production-ready. It focuses on issues including slow processing speeds and out-of-memory exceptions, and provides insights into the methods used to address these problems.