Kiran S.

Kiran S.

Data Engineer, bigdata architect

, United Kingdom

Experience: 16 Years

Kiran

Data Engineer, bigdata architect

36000 USD / Year

  • Immediate: Available

16 Years

Now you can Instantly Chat with Kiran!

About Me

I have 15+ years experience in IT.  5+ years experience in Biodata technologies (Apache spark, Kafka, Hadoop, CDH). Proficient experience in Java, Scala and Python programming languages

...

Show More

Portfolio Projects

Data Transformation

Company

Data Transformation

Description

Responsibilities:   
•    Analyse and design solution for various Corporate and retail CAIS (Banking, Credit cards, Loans), CCDS and CATO portfolios
•    Design and Created ingestion pipelines within spark which extract data files and stored in data lake (Hadoop) including profiling, cleaning, standardised data.
•    Design and developed near real-time data ingestion jobs which receive data from Kafka message queue and stored in data lake.
•    Created hive and impala tables in Data lake (Hadoop).
•    Transform data using spark with Scala and stored data in datalake.
•    Created bash (shell) scripts to run spark jobs and ingestion pipelines.
•    Design and implemented Jenkins pipelines for Continues integration and continues deployment (CICD)
•    Scheduled and monitored jobs in control-M

 

Show More Show Less

Cloud migration (Google Cloud)

Company

Cloud migration (Google Cloud)

Description

Responsibilities:   
•    Data warehouse architecture design in Google Cloud platform.
•    Design data migration plan to cloud environment from on-promise environment.
•    Build Google dataflow transformation jobs in Apache Beam.
•    Created tables in BigQuery underlined data stored in Google Cloud storage.
•    Scheduled and monitor jobs in Apache Airflow

 

Show More Show Less

Tools

Eclipse

Data Lake

Company

Data Lake

Description

Responsibilities:   
•    Responsible for Data Lake architecture design.
•    Requirements gathering from various teams (Loans, Credit cards, MI, etc.).
•    GAP analysis for EDIE ( Sainsbury's ETL framework and developed using Spark in Scala language)
•    Designed enhancement for EDIE to automate work which will reduce developer work and production job execution time.
•    Defined reconciliation rules for all layers in data lake and enhance EDIE to support reconciliation.
•    Defined folder structure for S3 and HDFS to manage security level 1-4 users groups.
•    Infrastructure build with Terraform.

 

Show More Show Less

Tools

Eclipse

CAVM

Company

CAVM

Description

Responsibilities:   
•    Architecture designed Hadoop with SAS based solution for CAVM project (including SAS Grid Manager for Hadoop, SAS Data Loader for hadoop, Enterprise Guide, Enterprise Miner, etc.)
•    Requirements gathering from various teams (Finance, credit, Data science, MI, ,etc.).
•    Designed Hybrid data lake (Hadoop & SQL server) architecture including polybase.
•    Designed plug and play architecture (Easily loose coupled), easily can migrate to new technologies.
•    Jobs are designed  simple and easy maintenance. Architecture designed with Generic jobs which will reusable for multiple use cases. Easily can switch to new technology with limited jobs development.
•    CAVM Platform designed with scalable platform (Process and storage)
•    Data Governance (Access Management, Data lineage, etc.)
•    Data Lake designed for Hadoop
•    Created data model for data warehouse (Fact and Dim) and Data Marts
•    Architecture supports to process social media and web data
•    Designed real time dash boards
•    Architecture designed high performance (Parallel execution)
•    Provided DaaS (Data As A Service) to integrate with existing web applications
•    Defined storage technologies, data flow and retention policies
•    Software installation and configuration management with Puppet.

 

Show More Show Less

Tools

ErWin
Share:

Verifications

  • Profile Verified

  • Phone Verified

Preferred Language

  • English - Fluent

Available Timezones

  • Eastern Daylight [UTC -4]

  • Mountain Daylight [UTC -6]

  • Pacific Daylight [UTC -7]

  • Central Daylight [UTC -5]

  • Eastern European [UTC +2]

  • Further EET [UTC +3]

  • Eastern EST [UTC +3]

  • Greenwich Mean [UTC ±0]

  • Australian EDT [UTC +11]

  • Australian CDT [UTC +10:30]