Responsibilities:
• Analyse and design solution for various Corporate and retail CAIS (Banking, Credit cards, Loans), CCDS and CATO portfolios
• Design and Created ingestion pipelines within spark which extract data files and stored in data lake (Hadoop) including profiling, cleaning, standardised data.
• Design and developed near real-time data ingestion jobs which receive data from Kafka message queue and stored in data lake.
• Created hive and impala tables in Data lake (Hadoop).
• Transform data using spark with Scala and stored data in datalake.
• Created bash (shell) scripts to run spark jobs and ingestion pipelines.
• Design and implemented Jenkins pipelines for Continues integration and continues deployment (CICD)
• Scheduled and monitored jobs in control-M

Responsibilities:
• Data warehouse architecture design in Google Cloud platform.
• Design data migration plan to cloud environment from on-promise environment.
• Build Google dataflow transformation jobs in Apache Beam.
• Created tables in BigQuery underlined data stored in Google Cloud storage.
• Scheduled and monitor jobs in Apache Airflow

Responsibilities:
• Architecture designed Hadoop with SAS based solution for CAVM project (including SAS Grid Manager for Hadoop, SAS Data Loader for hadoop, Enterprise Guide, Enterprise Miner, etc.)
• Requirements gathering from various teams (Finance, credit, Data science, MI, ,etc.).
• Designed Hybrid data lake (Hadoop & SQL server) architecture including polybase.
• Designed plug and play architecture (Easily loose coupled), easily can migrate to new technologies.
• Jobs are designed simple and easy maintenance. Architecture designed with Generic jobs which will reusable for multiple use cases. Easily can switch to new technology with limited jobs development.
• CAVM Platform designed with scalable platform (Process and storage)
• Data Governance (Access Management, Data lineage, etc.)
• Data Lake designed for Hadoop
• Created data model for data warehouse (Fact and Dim) and Data Marts
• Architecture supports to process social media and web data
• Designed real time dash boards
• Architecture designed high performance (Parallel execution)
• Provided DaaS (Data As A Service) to integrate with existing web applications
• Defined storage technologies, data flow and retention policies
• Software installation and configuration management with Puppet.

Responsibilities:
• Responsible for Data Lake architecture design.
• Requirements gathering from various teams (Loans, Credit cards, MI, etc.).
• GAP analysis for EDIE ( Sainsbury's ETL framework and developed using Spark in Scala language)
• Designed enhancement for EDIE to automate work which will reduce developer work and production job execution time.
• Defined reconciliation rules for all layers in data lake and enhance EDIE to support reconciliation.
• Defined folder structure for S3 and HDFS to manage security level 1-4 users groups.
• Infrastructure build with Terraform.