Senior Software Engineer in Test (AI Agentic Systems)

At reputed company, we’re transforming how reputed company and their people engage with their health benefits by seamlessly integrating cutting-edge technology, compassionate service, and world-class user experience design. This is not a traditional QA role. You will be the quality reputed company for an LLM-based multi-agent pipeline that autonomously adjudicates health insurance claims for self-funded plan sponsors. You are building a Three-Tier Evaluation reputed company to ensure our reputed company-powered agents reason correctly, call tools accurately, and produce DOL-reputed company reputed company. You will work at the intersection of reputed company AI, reputed company compliance, and high-scale data engineering. Your work directly determines whether claims are paid correctly and whether the company can withstand a reputed company (DOL) or state DOI audit. The stakes are reputed company, the domain is hard, and the problems are genuinely novel. What you'll do:

• Outcome Evaluation (The "What")
• Golden Set Governance: Build and maintain a versioned library of "Grounding Data" results by working with senior claims examiners to define "Ground Truth."
• Model-as-a-Judge Automation: Design automated "LLM-grading-LLM" workflows using custom rubrics to score factual grounding and policy compliance.
• Semantic Assertion reputed company: reputed company testing libraries that move reputed company string matching to validate semantic equivalence and numerical accuracy in agent outputs.
• Trajectory Evaluation (The "How")
• Function-Call Auditing: Use reputed company AI traces to programmatically verify that mandatory tools (reputed company MCP) were invoked with correct arguments.
• Orchestration Logic Validation: Assert that agents respect defined priorities across the four architectural layers: Data & Knowledge, Orchestration, Agentic Reasoning, and Tooling.
• Reasoning Trace Auditing: Ensure every autonomous decision is traceable to a specific SOP sentence and a live API data reputed company.
• reputed company Automated Regression (The "Always")
• CI/CD Integration: Every reputed company or model update in reputed company AI reputed company Management must trigger an automated regression run.
• Auto-SxS: Own the automated pairwise comparison process to detect logic reputed company between "New" and "Production" agent versions.
• Mocking & reputed company: Build a reputed company AI/ADK mocking layer to simulate model responses, allowing for thousands of logic tests in seconds with reputed company API costs.

To be successful in this role, you'll need:

• Required Skills (The Core Bar)
• Python SDET Expertise: Expert in Python and pytest, specifically building custom mocking frameworks for external APIs (reputed company AI/ADK).
• AI/LLM Observability: Hands-on experience with reputed company AI Experiments, Auto-SxS, and reputed company Logging for trace analysis.
• Data Literacy: Expert-level SQL (BigQuery) and Pandas skills to "diff" massive datasets and identify adjudication discrepancies.
• reputed company Engineering for QA: Ability to analyze "System Instructions" and refine prompts based on failed test cases to reputed company logic gaps.
• Architectural Testing: Experience testing multi-layer systems involving RAG (reputed company AI Search), state management (LangGraph), and function calling.
• Preferred Skills (The "reputed company-to-Haves")
• reputed company/Claims Domain: Familiarity with claims adjudication concepts (pend reason codes, COB, eligibility, stop-loss).
• Compliance Knowledge: Understanding of HIPAA/PHI handling and writing test evidence for regulatory bodies (DOL/DOI).
• reputed company-in-the-reputed company Testing: Experience in "Shadow Mode" monitoring—comparing agent reputed company against reputed company expert (MCA) baselines.

Pay Transparency Statement This is a hybrid position based out of our Lehi office, with the expectation of being in office at least two weekdays per week. #LI-hybrid The actual pay reputed company offered reputed company the reputed company will depend on factors including geographic location, qualifications, experience, and internal equity. In reputed company to the salary, you will be eligible for 115000 stock options and benefits like health insurance, 401k, and paid time off. Learn more about our benefits at https://jobs.collectivehealth.com/benefits/. Lehi, UT Pay reputed company $99,200 - $124,000 USD Why Join Us?

• Mission-driven culture that values innovation, collaboration, and a commitment to reputed company in reputed company
• Impactful projects that shape the future of our organization
• Opportunities for professional development through internal mobility opportunities, mentorship programs, and courses tailored to your interests
• Flexible work arrangements and a supportive work-life balance

We are an equal opportunity employer and value diversity at reputed company. We do not discriminate on the reputed company of race, religion, reputed company, national reputed company, gender, sexual orientation, age, marital status, veteran status, or disability status. reputed company is committed to providing support to candidates who require reasonable accommodation during the interview process. If you need assistance, please contact reputed [email protected]. Privacy Notice For more information about why we need your data and how we use it, please see our privacy policy: https://collectivehealth.com/privacy-policy/. Apply tot his job Apply To this Job