Applied AI Software Engineer

Job Summary

A company is looking for an Applied AI Software Engineer to lead evaluations for AI agents in development and post-deployment.

Key Responsibilities

• Design and execute large-scale evaluation plans for LLM-based agents in various operational tasks
• Build end-to-end test harnesses to validate model behavior under different configurations
• Analyze results and summarize tradeoffs for product and engineering stakeholders

Required Qualifications

• 5+ years of experience in applied machine learning or AI engineering, focusing on evaluation and benchmarking
• Proficiency with foundation model APIs and experience with complex agent behaviors
• Experience designing and running high-throughput evaluation pipelines
• Strong Python engineering skills and familiarity with data engineering tools
• Familiarity with clinical or healthcare data is a strong plus