Responsibilities:   
•    Analyse and design solution for various Corporate and retail CAIS (Banking, Credit cards, Loans), CCDS and CATO portfolios
•    Design and Created ingestion pipelines within spark which extract data files and stored in data lake (Hadoop) including profiling, cleaning, standardised data.
•    Design and developed near real-time data ingestion jobs which receive data from Kafka message queue and stored in data lake.
•    Created hive and impala tables in Data lake (Hadoop).
•    Transform data using spark with Scala and stored data in datalake.
•    Created bash (shell) scripts to run spark jobs and ingestion pipelines.
•    Design and implemented Jenkins pipelines for Continues integration and continues deployment (CICD)
•    Scheduled and monitored jobs in control-M

Responsibilities:   
•    Data warehouse architecture design in Google Cloud platform.
•    Design data migration plan to cloud environment from on-promise environment.
•    Build Google dataflow transformation jobs in Apache Beam.
•    Created tables in BigQuery underlined data stored in Google Cloud storage.
•    Scheduled and monitor jobs in Apache Airflow

Responsibilities:   
•    Responsible for Data Lake architecture design.
•    Requirements gathering from various teams (Loans, Credit cards, MI, etc.).
•    GAP analysis for EDIE ( Sainsbury's ETL framework and developed using Spark in Scala language)
•    Designed enhancement for EDIE to automate work which will reduce developer work and production job execution time.
•    Defined reconciliation rules for all layers in data lake and enhance EDIE to support reconciliation.
•    Defined folder structure for S3 and HDFS to manage security level 1-4 users groups.
•    Infrastructure build with Terraform.

Responsibilities:   
•    Architecture designed Hadoop with SAS based solution for CAVM project (including SAS Grid Manager for Hadoop, SAS Data Loader for hadoop, Enterprise Guide, Enterprise Miner, etc.)
•    Requirements gathering from various teams (Finance, credit, Data science, MI, ,etc.).
•    Designed Hybrid data lake (Hadoop & SQL server) architecture including polybase.
•    Designed plug and play architecture (Easily loose coupled), easily can migrate to new technologies.
•    Jobs are designed  simple and easy maintenance. Architecture designed with Generic jobs which will reusable for multiple use cases. Easily can switch to new technology with limited jobs development.
•    CAVM Platform designed with scalable platform (Process and storage)
•    Data Governance (Access Management, Data lineage, etc.)
•    Data Lake designed for Hadoop
•    Created data model for data warehouse (Fact and Dim) and Data Marts
•    Architecture supports to process social media and web data
•    Designed real time dash boards
•    Architecture designed high performance (Parallel execution)
•    Provided DaaS (Data As A Service) to integrate with existing web applications
•    Defined storage technologies, data flow and retention policies
•    Software installation and configuration management with Puppet.