AI Engineer (Evaluations) - Hawthorn East at Coles Group in Melbourne, VIC

  • Company: Coles Group
  • Location: Hawthorn East Melbourne, VIC, Australia
  • Job type: full time
  • Workplace: onsite
  • Posted: 2026-11-19

Apply for this role

All open roles at Coles Group

Job description

We’ve been proudly serving Australian communities since 1914 and have grown to become a top 30-listed company on the ASX with a portfolio of iconic homegrown brands. At the heart of our continued success is over 115,000 team members working across the country, all contributing to bringing our customers smiles. You’ll not only get to see your work make a difference, you’ll be part of a team working together to help Australians eat and live better every day. About the team Our Technology team is the backbone of the business, building and maintaining the information technology infrastructure that support Coles Group. As a part of the Technology team, you get the opportunity to work through complex, challenging, meaningful problems using large data sets and new technologies to expand your skills and experience. About the role As the AI Engineer (Evaluations) in Coles’ AI Commerce team, you’ll be responsible for how we measure the quality of the AI agent in the Coles website and apps that helps customers find products, build their basket, manage orders and sort out issues. This is an AI engineering role focused on evaluation. You’ll build and manage our evals, keep them trustworthy, and turn the results into clear actions for the team. When the agent’s quality shifts, you’ll be the first to know why. This is a hands-on role. You’ll work AI-first, using agentic development tools to ship high-quality work at pace. You’ll work closely with the Technical Lead, AI engineers, product managers and designers to make sure what we build balances UX, quality and safety. If you want to shape the quality of a production AI agent used by customers across the nation, with the scale and backing of a major retailer behind you, this is that role. Key responsibilities will include: Build and maintain the evaluation framework for the AI Commerce Agent: golden datasets, test scenarios and LLM-as-judge metrics Run evals against every significant change and make the results a clear go/no-go signal for release Analyse results to find failure patterns, root causes and regressions, and turn them into clear, prioritised actions for engineers and product Calibrate LLM judges against human review so the scores can be trusted Grow coverage as the agent grows, including edge cases, safety and adversarial (red-team) scenarios Help build online evaluation from production conversations, tracing and customer feedback Tie eval metrics to product success metrics and report on agent quality to the team and stakeholders Use agentic development tools to ship high-quality code at pace Stay across developments in AI evaluation and bring what’s useful into our approach About you and your skills: We're looking for a sharp, curious engineer who cares about getting quality right. You're rigorous with data, you don't trust a number until you understand it, and you can explain what the results mean and what to do next. You take ownership of your work, learn quickly and back your views with evidence. You work well in a team and you're comfortable pushing back when the data says something isn't ready. You'll bring: Proven (2 to 3 years) experience in AI engineering, ML or data science, including hands-on work evaluating LLM or agentic systems Strong Python and SQL Hands-on experience with LLM evaluation: golden datasets, LLM-as-judge, rubric design and tracing A solid grounding in statistics and experiment design, such as sampling, significance and inter-rater agreement An understanding of how AI agents work: tool use, orchestration, RAG and guardrails AI-first ways of working. Agentic coding tools are core to how you build, and you know how to use them without compromising quality Clear communication. You can turn complex results into a story that engineers and stakeholders act on Pragmatism. You apply the 80/20 rule and know when good enough is right for now and when quality is non-negotiable What will make you stand out: Experience with red-teaming or AI safety testing Experien

AI Engineer (Evaluations) - Hawthorn East on JobPost.