Data Engineer (DATABRICKS / PYSPARK) at Sigma Software

Job description

  Develop and maintain data pipelines for processing large volumes of data in batch and near real-time environments Work with Databricks, PySpark, and SQL to transform, validate, and optimize data Support implementation of Data Lakehouse solutions following engineering standards and best practices Build, test, and automate ETL processes and data workflows Improve data quality, organization, and reliability across platforms Collaborate with engineers, analysts, and stakeholders to deliver scalable solutions Troubleshoot, optimize, and monitor data pipelines Stay up to date with modern data engineering technologies Ensure deliverables meet quality standards and project requirements 2+ years of experience in Big Data development and database design Strong hands-on experience with SQL Strong Python development skills Hands-on experience with PySpark Experience developing and maintaining data pipelines in Databricks Experience with Delta Lake Experience with Medallion Architecture (Bronze, Silver, Gold layers) Strong analytical and problem-solving skills Intermediate or higher English level WILL BE A PLUS Experience with AWS services, including S3