Human Data Quality Analyst, AI Business at Prolific

Job description

Human Data Quality Analyst, AI Business Prolific Prolific isn’t just enabling AI innovation – we’re redefining it. While foundational AI technologies are becoming commoditized, Prolific’s human data infrastructure provides the high-quality, diverse data required to train the next generation of AI models. Through our platform, we empower researchers and companies to access a global, ethically curated participant base, ensuring cutting-edge AI research and training grounded in inclusivity and precision. The Role Prolific provides the human data that powers the next generation of AI models, working with frontier labs to capture the complex human judgments researchers need to train, evaluate and improve them. As a Human Data Quality Analyst, you'll be on the front line of making sure that data captures the right signal and is genuinely good enough to do its job. This isn’t traditional, back-office QA. You’ll be doing real analytical work: digging into datasets, identifying patterns and failure modes, investigating why quality has shifted, and turning complex findings into clear insights that help us improve how data is collected, reviewed and delivered. You'll spend real time reading annotations closely, but that is how you gather evidence, not what you produce. What you produce is analysis, practical recommendations and better quality controls. You'll get hands-on exposure to human data, annotation, machine learning pipelines and AI evaluation, working alongside Quality, Engineering, Operations and Delivery on new and evolving problems. There won't always be an established playbook. You'll be guided by our quality engineers, but you'll also need to run your own analysis, test your assumptions and recognise when you need input. It is a role with a steep learning curve from day one and a strong opportunity for someone early in their career to build deep, practical experience in a fast-moving area of AI. What You’ll Be Doing Run day-to-day quality measurement across live human-data programmes, combining statistical analysis with regular hands-on review to determine whether the data is fit for its intended use. Find the signal behind quality issues by investigating changes, quantifying their impact and distinguishing individual errors from wider problems in guidance, task design, training or tooling. Help build the data-quality pipeline alongside our quality engineers, turning repeated analysis into reliable validations and automated systems, including rule-based, LLM, and ML assisted approaches. Turn complex analysis into a clear story through reporting and visualisations that help stakeholders understand what is happening, why it matters, and what action the evidence supports. Support new programmes from design through to launch by helping develop the rubrics, guidelines, gold sets, calibration exercises, and production gates to measure quality from the start. Use what we learn from live programmes to improve annotation design, guidance, training and quality processes, then measure whether those changes work. What You’ll Bring to the Role You have 1 to 2 years of practical experience analysing real data in an analytical, quality, research or data-focused role. You can use SQL and Python to write your own queries and analysis, check whether the results make sense, and explain how you reached them. You can apply statistical concepts such as sampling, distributions and variability, and judge whether a result reflects a meaningful pattern or limited evidence. You can turn evidence into a clear account of what is happening, supported by concrete examples and practical recommendations that non-technical colleagues can act on. You can assess detailed work carefully and consistently while looking for patterns across the wider dataset, process or programme. When the right method is unclear, you can test a sensible approach, document your reasoning and recognise when you need input. Even Better if you have Experience working with data produc