Claude Skills Evaluation & Regression Testing with Snowflake Cortex Agent

A developer on r/ClaudeAI has deployed a Claude credit risk agent sitting on top of Snowflake Cortex Agent with a semantic layer. The agent is in production and getting positive feedback, but the real challenge is maintaining and upgrading it — specifically, regression and evaluation of small changes to skills.

Current Setup

• Semantic model and data foundation already in place (years of investment)
• Production-grade observability available in Snowflake for potential automation
• For testing, the team manually evaluates agent results against existing BI queries

The Problem

The developer notes that most articles on this topic are generic and written by people who haven't actually shipped to production. They're looking for others working on similar problems in the trenches, specifically around:

• Automated evaluation of analytics AI/BI agent outputs
• Regression testing when skills are updated
• Leveraging Snowflake observability for test automation

If you're building evaluation pipelines for AI analytics agents, the discussion thread has comments from others in similar situations.