About Me
Experienced Data Engineer with a demonstrated history of working in the information technology and services industry, Skilled in Big Data Analytics in AWS, ELK Stack - PySpark, Scala, Apache Nifi, Scala, Airflow. Strong Engineering background wi...
Show MoreSkills
Others
Web Development
Software Testing
Data & Analytics
Software Engineering
Database
Programming Language
Development Tools
Positions
Portfolio Projects
Company
Cloud Data Migration and Analysis
Description
● A complete end to end solution for organizations needing to
utilize the power of cloud and data analytics.
● Extracting valuable information from the data in a scheduled and
automated manner.
● The data is processed using spark and scala and stored in various
DB’s across pipelines after finishing and changing data into
required format.
● AWS S3 was used for Cloud Storage. Designed Monitoring and
Alerting Platform using AWS Lambda, Apache NiFi, ELK, Python
and SQS.
Company
Data Analysis and Processing for Health Insurance Company
Description
- The platform integrates all the data sources, internal and external to the company, to the Data Lake.
- It stores all kinds of data and makes it available immediately to different business and development teams for their work.
- The data is processed using spark scala and stored in s3. The Platform was built on a serverless infrastructure (Lambda), thus optimizing cost and enabling auto-scaling capabilities.
- AWS services like S3, EMR, KMS/IAM were used and implemented in the architecture to provide reliable storage,
compute and security.
Company
Data Ingestion and Processing of Credit Bureau Data
Description
- The platform migrates data from different sources to Cloud and processes data using Apache Spark.
- Reading blob data using spark and ingesting it and storing it in various DB’s across pipelines after finishing and changing data into the required format. AWS was used for Cloud
- ELK for Monitoring and Logging Platform.
Tools
sublime (linux) IntelliJ IDEACompany
Data Analysis and Processing for Call Allocation Engine
Description
- A complete end to end solution using a combination of Data Engineering and Data Science for Call Allocation of service
engineers. - Migration of data from on-prem to Cloud using Aws DMS and using spark scala for data processing and view the data in Athena using AWS Glue, Process data using S3 to another bucket using Lambda function by adding jobs as steps in EMR, Visualizing data using Tableau.