Senior Data Engineer at BP in Kuala Lumpur

Job description

Entity: Technology Job Family Group: IT&S Group Job Description: Let me tell you about the role At the heart of bp's technological endeavors is the Technology organization. We are the driving force behind the technology that fuels bp's operations, from upstream energy production to downstream delivery to our esteemed customers. Our diverse teams cater to a broad spectrum of interests, encompassing everything from infrastructure and backend services to customer-centric web and native applications. With a keen emphasis on agility, we foster using native AWS and Azure services as our infrastructure foundation, including serverless architectures. Our ethos is not just about using the best tech for the task but also encouraging an environment where our software and platform engineers thrive, learn, and continuously evolve. Key Responsibilities Drive engineering best practices through code reviews, technical standards, CI/CD practices, and automation Lead source control, release management, and deployment processes for data engineering solutions. Facilitate technical discussions, architecture reviews, and technology decision-making across teams. Mentor engineers and contribute to building a high-performing engineering culture through knowledge sharing and technical leadership. Ensure compliance with information security, governance, and operational standard methodologies. Support the reliability, resilience, and continuous improvement of data platforms and services. Design, build, and operate scalable, secure, reliable, and efficient data products, pipelines, and platforms. Own the end-to-end lifecycle of data solutions, ensuring operational accountability, supportability, and continuous improvement. Collaborate with business stakeholders, engineers, architects, analysts, and data scientists to deliver high-quality data solutions. Design and maintain data models, ingestion frameworks, and transformation pipelines that ensure data quality, integrity, and performance. Implement DataOps best practices including CI/CD, Infrastructure as Code, automated testing, monitoring, and deployment automation. Build observability, alerting, and automated data quality controls to ensure platform reliability and rapid issue detection. Maintain metadata, lineage, documentation, runbooks, and support processes to improve governance and operational readiness. Lead incident resolution, root cause analysis, and reliability improvements to reduce recurring issues and improve service outcomes. Ensure compliance with security, privacy, governance, and enterprise architecture standards. Provide technical leadership through architecture guidance, mentoring, code reviews, and engineering best practices, including good team work What you will need to be successful (experience and qualifications) 8+ years of experience in Data Engineering, Software Engineering, or related disciplines. 5+ years of hands-on Python development experience. 3+ years of experience with PySpark and large-scale distributed data processing. Shown experience delivering and operating enterprise-scale data platforms and pipelines. Experience working in DataOps, DevOps, SRE, or production support environments. Technical Skills Strong expertise in SQL, database design, and data modelling. Experience designing end-to-end data architectures including ingestion, transformation, storage, governance, and consumption layers. Strong AWS experience, including services such as S3, Glue, Lambda, Athena, Redshift, and EventBridge. Hands-on experience with Databricks and modern Lakehouse architectures. Experience with Git, CI/CD, Infrastructure as Code, and deployment automation. Operational Excellence Strong understanding of observability, monitoring, alerting, and platform reliability. Experience implementing data quality controls, lineage, metadata, and governance practices. Experience with incident management, root cause analysis, and continuous service improvement. Knowledge of security, privacy, co