Staff Engineer – AI Data Engineering at Prudential in Group Head Office

Job description

Prudential’s purpose is to be partners for every life and protectors for every future. Our purpose encourages everything we do by creating a culture in which diversity is celebrated and inclusion assured, for our people, customers, and partners. We provide a platform for our people to do their best work and make an impact to the business, and we support our people’s career ambitions. We pledge to make Prudential a place where you can Connect, Grow, and Succeed. We are looking for an experienced AI Data Engineering Lead to define and lead the engineering of enterprise-scale data platforms and data products that power Artificial Intelligence, Machine Learning, Generative AI and advanced analytics. This individual will build the organisation’s AI-ready data foundation by bringing together modern data engineering, cloud platforms, batch and real-time processing, unstructured data, vector and semantic technologies, data governance, and AI/ML engineering practices. The role requires a senior technology leader with 15+ years of experience, strong hands-on technical depth, architecture capability, and a proven ability to lead engineering teams and deliver complex data platforms at enterprise scale. Key Responsibilities AI Data Engineering Strategy & Architecture · Define the enterprise AI Data Engineering strategy, architecture and engineering standards. · Design scalable data architectures supporting AI/ML, GenAI, LLM, analytics and enterprise data products. · Establish reusable architectural patterns for structured, semi-structured and unstructured data. · Define patterns for data lakes, lakehouse architectures, data warehouses, streaming platforms, vector databases and AI knowledge layers. · Drive architectural decisions covering performance, scalability, resiliency, security and cost. AI-Ready Data Platforms · Lead engineering of data platforms providing trusted and reusable data for ML models, LLMs, AI agents and GenAI applications. · Build ingestion and transformation frameworks for data from databases, APIs, documents, events, logs and enterprise applications. · Establish pipelines for batch, near-real-time and streaming data processing. · Drive automated data validation, quality controls, metadata management and lineage. · Establish engineering approaches for preparing text, documents, images and other unstructured data for AI consumption. GenAI & LLM Data Engineering · Provide technical leadership for Retrieval-Augmented Generation (RAG), document ingestion, parsing, chunking and enrichment. · Lead embedding generation and management, vector and hybrid search, semantic retrieval and reranking. · Design knowledge repositories, metadata and access-control filtering, and enterprise search integration. · Create data pipelines for LLM evaluation, AI grounding, authorised retrieval, fine-tuning where applicable, and AI agents. · Understand how poor retrieval, incomplete or stale data, weak metadata and inappropriate chunking affect enterprise AI reliability. Modern Data Engineering · Provide leadership across ETL/ELT, distributed processing, lakehouse and data warehouse architectures, streaming, event-driven design, data APIs and data services. · Drive data modelling, data contracts, observability, quality engineering, lineage and metadata management. · Embed Infrastructure-as-Code, CI/CD, DevOps and DataOps practices in the engineering lifecycle. · Improve reusability, automation, reliability and developer productivity through common frameworks and standards. Cloud & Technology Platforms · Demonstrate significant expertise across modern cloud and data platforms such as Microsoft Azure, Azure Databricks, Microsoft Fabric, Azure Data Factory, Azure Synapse Analytics, Azure Data Lake Storage, Snowflake and Databricks. · Bring strong capability in Python, SQL, Apache Spark/PySpark, Kafka, Delta Lake, Airflow and dbt. · Apply AI and GenAI data technologies including Azure AI Search, vector databases, embedding models, semantic search,