Senior Data Engineer  

About the Role 

The Senior Data Engineer in our AI & Data team will be responsible for designing and building 

scalable data platforms, enterprise-grade data architectures, and high-performance data 

ingestion frameworks across Azure, Snowflake, Databricks, and Lakebase. 

This role requires a highly technical engineer capable of solving complex data platform 

challenges involving large-scale API integrations, distributed data processing, cloud-native 

architectures, and AI-enabled data platforms. 

The ideal candidate is not only a Data Engineer but also a strong Python engineer with expertise 

in data architecture, system design, API engineering, concurrent processing, and enterprise

scale platform development.  

Main Responsibilities 

• Design and implement scalable enterprise data architectures supporting AI, analytics, 

reporting, and operational workloads. 

• Build and optimize large-scale ELT/ETL pipelines using Databricks, Snowflake, Azure Data 

Factory, and Azure services. 

• Design and implement Medallion Architectures (Bronze/Silver/Gold), CDC frameworks, 

lineage models, and master data management workflows. 

• Develop high-performance Python-based ingestion frameworks supporting large-scale 

extraction from internal and external data sources. 

• Build and maintain REST API and GraphQL integrations including OAuth/OAuth2 

authentication, token lifecycle management, pagination, retry mechanisms, rate limiting, 

and automated error recovery. 

• Design parallelized ingestion solutions using multi-threading, asynchronous processing, and 

distributed execution patterns. 

• Develop and maintain Databricks pipelines, Delta Lake architectures, and Snowflake 

analytical data platforms. 

• Build and support real-time and near-real-time data processing solutions using event-driven 

architectures. 

• Implement data quality, reconciliation, lineage, observability, and governance frameworks 

across the platform. 

• Design scalable data models supporting analytics, machine learning, feature engineering, 

and AI workloads. 

• Monitor production environments, troubleshoot pipeline failures, optimize platform 

performance, and drive cost optimization initiatives. 

• Collaborate closely with AI Engineers, Architects, Product Teams, and Business Stakeholders. 

Required Skills & Experience 

Python (Mandatory) 

• Strong production-grade Python development experience. 

• Object-Oriented Programming (OOP). 

• Modular framework development. 

• Logging, exception handling, testing, and debugging. 

• Performance optimization and profiling. 

• Experience building reusable ingestion and transformation frameworks. 

Advanced API Engineering (Mandatory) 

• REST APIs. 

• GraphQL APIs. 

• OAuth/OAuth2. 

• JWT Authentication. 

• API Pagination. 

• Rate Limiting. 

• Retry Logic. 

• Token Refresh Handling. 

• Error Handling Frameworks. 

• High-volume API ingestion architecture. 

Data Architecture & System Design (Mandatory) 

• Medallion Architecture. 

• Data Warehouse Architecture. 

• Data Lakehouse Architecture. 

• Master Data Management (MDM). 

• Golden Record Design. 

• Data Lineage. 

• Change Data Capture (CDC). 

• Historical Data Management. 

• Enterprise Data Modeling. 

Data Pipeline Design (Mandatory)  

• Design and build scalable, fault-tolerant enterprise data pipelines.  

• Strong experience with batch, near real-time, and event-driven processing architectures. 

• Expertise in designing ingestion, transformation, validation, reconciliation, and serving 

layers across modern data platforms.  

• Experience implementing Medallion Architecture (Bronze, Silver, Gold) and data lakehouse 

patterns.  

• Strong understanding of Change Data Capture (CDC), incremental processing, watermarking 

strategies, and SCD Type 1/Type 2 implementations.  

• Design audit frameworks, lineage tracking, reconciliation controls, monitoring, alerting, and 

observability solutions.  

• Ability to build high-volume ingestion pipelines from APIs, databases, files, data streams, 

and external systems.  

• Experience designing resilient pipelines with retry mechanisms, checkpointing, idempotent 

processing, restartability, and failure recovery.  

• Strong understanding of throughput optimization, parallel processing, concurrency, and 

workload orchestration.  

• Experience designing data movement patterns across Snowflake, Databricks, Azure services, 

and downstream analytics platforms. 

Databricks (Mandatory) 

• Databricks Workflows. 

• Delta Lake. 

• Unity Catalog. 

• PySpark. 

• Spark SQL. 

• Databricks Performance Tuning. 

• Distributed Data Processing. 

Snowflake (Mandatory) 

• Snowpipe. 

• Streams. 

• Tasks. 

• Dynamic Tables. 

• Time Travel. 

• Zero-Copy Cloning. 

• RBAC. 

• Query Optimization. 

• Warehouse Optimization. 

• Cost Management. 

SQL & Data Modeling (Mandatory) 

• Advanced SQL. 

• CTEs. 

• Window Functions. 

• Stored Procedures. 

• MERGE. 

• Star Schema. 

• Snowflake Schema. 

• SCD Type 1 & Type 2. 

• Dimensional Modeling. 

Azure (Mandatory) 

• Azure Data Factory. 

• ADLS Gen2. 

• Azure Synapse. 

• Event Hubs. 

• Azure Integration Services. 

Strongly Preferred 

• Apache Airflow. 

• dbt. 

• Kafka. 

• Event Driven Architecture. 

• Terraform. 

• Infrastructure as Code. 

• Azure OpenAI. 

• AI/ML Data Pipelines. 

• Feature Stores. 

Git & DevOps (Mandatory) 

• Git Branching Strategies. 

• Pull Requests. 

• Git Rebase. 

• Cherry-picking. 

• Merge Management. 

• Repository Governance. 

• CI/CD Pipelines. 

• GitHub Actions / Azure DevOps. 

• Secret Management. 

• Git History Cleanup and Recovery. 

Experience 

• 5+ years of hands-on Data Engineering experience. 

• Proven experience designing production-grade enterprise data platforms. 

• Strong experience working directly with business stakeholders and translating requirements 

into scalable technical solutions. 

• Experience leading architecture discussions and solving complex technical problems 

independently.