Job Summary:

We are seeking a skilled Data Engineer with hands-on experience in Databricks and Big Data technologies to join our growing data team. The ideal candidate will be responsible for designing, building, and maintaining scalable data pipelines, working closely with data scientists, analysts, and business stakeholders to ensure high-quality, accessible data.

Key Responsibilities:

• Design, implement, and maintain robust, scalable data pipelines and ETL processes using Apache Spark on Databricks.
• Develop and optimize Delta Lake architecture and data workflows.
• Collaborate with cross-functional teams to understand business requirements and translate them into data solutions.
• Implement and monitor data quality checks and validation routines.
• Ensure data governance, security, and compliance with industry standards.
• Automate data workflows using Databricks Workflows, Airflow, or other orchestration tools.
• Work with structured and unstructured data, leveraging cloud-native services (e.g., Azure Data Lake, AWS S3, etc.).
• Optimize Spark jobs for performance and cost efficiency in the Databricks environment.

Required Skills and Qualifications:

• 3+ years of experience as a Data Engineer or similar role.
• Strong experience with Databricks and Apache Spark (Scala or PySpark).
• Hands-on experience with Delta Lake and Lakehouse architecture.
• Proficient in writing complex SQL queries and working with large-scale data systems.
• Experience with cloud platforms (Azure, AWS, or GCP).
• Familiarity with CI/CD pipelines and DevOps practices for data workflows.
• Strong understanding of data modeling, data warehousing, and data lake concepts.
• Excellent communication and collaboration skills.