Back to Search Results

Vishal J.

Data Engineer III

Commitment
0/ 5
Competency
0/ 5
Reliability
0/ 5
  • Overall Experience: 7 Years  
  • Amazon Relational Database Service:
  • Apache Spark:
  • API Development:
  • Artificial Intelligence:
  • DATAPROC:
+ More

Vishal J. 

Data Engineer III

Commitment
0/5
Competency
0/5
Reliability
0/5

Time zones ready to work

Willing to travel to client location: Yes  

About Me 

Data Engineer
Data Engineer

Interview Videos

Signup to see videos

Risk-Free Trial, Pay Only If Satisfied.

Portfolios

Document Extractor app

Website Link:

Role:

Extract the data from the pdf le and store this data into the database. Later visualize this data UI This project was running on falsk application le. via UI user can upload le into s3 bucket then lambda gets triggered and data get extracted from upload les and store it into RDS database.
Extract the data from the pdf le and store this data into the database. Later visualize this data UI This project was running on falsk application le. via UI user can upload le into s3 bucket then lambda gets triggered and data get extracted from upload les and store it into RDS database.

Skills: PythonFlaskAWS S3AWS LambdaPDF

Tools:

Document digitization

Website Link:

Role:

Document digitization

  • extracting data from pdf and convert it into different format such as JSON, XML, csv as per customer requirement
  • building end to end pipelines
  • building automation piplines for reporting

Document digitization

  • extracting data from pdf and convert it into different format such as JSON, XML, csv as per customer requirement
  • building end to end pipelines
  • building automation piplines for reporting

Skills: AWS ECSRemote Desktop Services - (RDS)DockerPandasLinux

Tools:

Employment engagement service

Website Link:

Role:

Design a database schema, Writing models, view/functionality for the application

● Hosing the application on AWS - ec2 windows instance, RDS.

Design a database schema, Writing models, view/functionality for the application

● Hosing the application on AWS - ec2 windows instance, RDS.

Skills: PythonAWS RDSSQLDjango

Tools:

Predictive works (predictive analytics)

Website Link:

Role:

The aim of the project is to predict the major incident before occurring.

● Data Collection from different databases

● Writing an ETL script using python.

● Creating an end to end data pipelines.

● Hosting a project in a cloud with Aws services like- S3, ec

The aim of the project is to predict the major incident before occurring.

● Data Collection from different databases

● Writing an ETL script using python.

● Creating an end to end data pipelines.

● Hosting a project in a cloud with Aws services like- S3, ec2, Lambda, Batch.

● Building a Docker image for AWS Batch

● Design a Memory management module using python for AWS lambda.

Show More

Skills: ETL(Extract, Transform, Load)AWS S3AWS LambdaPostgreSQLAWS EC2

Tools:

Alerting system for a production system

Website Link:

Role:

Setup the ELK Stack on the production system.

● Building a single and multimeric machine learning job using Elasticsearch, Logstach, and kibana.

Setup the ELK Stack on the production system.

● Building a single and multimeric machine learning job using Elasticsearch, Logstach, and kibana.

Skills: ELK Stack

Tools:

+ More

Employment

Data Engineer

2019/11 -

Skills: Remote Desktop Services - (RDS)DockerAWS S3PythonAWS ECSGitAWS ApiGateWayAWS RDSFlask

Your Role and Responsibilities:

Creating an end to end data pipelines for clients

● Writing script and AWS lambda code using python

● Working with AWS technologies like S3, EC2, ECR, ECS, Lambda,API gateway, RDS, PostgreSQL, Docker

● Production deployment

Creating an end to end data pipelines for clients

● Writing script and AWS lambda code using python

● Working with AWS technologies like S3, EC2, ECR, ECS, Lambda,API gateway, RDS, PostgreSQL, Docker

● Production deployment

Jr. Data Engineer

2018/02 - 2019/10

Skills: ETL(Extract, Transform, Load)AWS LambdaAWS EC2PostgreSQLMySQLDjango

Your Role and Responsibilities:

The data collection from different systems.

● Data analysis and Building a Machine Learning Job (single and multimetric)in ELK stack

. ● Developed a Python Script for ETL job Which downloads Application Log Data from LogicMonitor and Store in S3 bucket.

● Work with A

The data collection from different systems.

● Data analysis and Building a Machine Learning Job (single and multimetric)in ELK stack

. ● Developed a Python Script for ETL job Which downloads Application Log Data from LogicMonitor and Store in S3 bucket.

● Work with Aws EC2, Lambda, Batch, and cloud formation.

● Developed a Python Memory Management Module for Aws Lambda.

Show More

Data Engineer

2019/09 -

Skills: PythonAWS S3AWS LambdaAWS SQSPostgreSQLPDF

Your Role and Responsibilities:

Building, Maintaining and optimizing end-to-end ETL data pipelines. Automated ETL process across thousand of pdf les which reduced manual workload by 85% Designed the data pipeline architecture that quickly scaled from 0 to 115000 for pdf le processing. Maintained data pipeline up-time of 99.9% whic
Building, Maintaining and optimizing end-to-end ETL data pipelines. Automated ETL process across thousand of pdf les which reduced manual workload by 85% Designed the data pipeline architecture that quickly scaled from 0 to 115000 for pdf le processing. Maintained data pipeline up-time of 99.9% which ingesting streaming data from di erent data sources using s3, Lambda, SQS, ECS, Postgresql, and Python. Decreased database load by ~25% by refactoring exiting python scirpts. Handling the production deployments
Show More

Data Engineer III

2022/06 - 2025/02

Skills:

Your Role and Responsibilities:

Designing building and optimizing data pipelines for efficient data processing and integration.Migrated 5TB+ of data across platforms ensuring seamless integration and performance.Performed data migration and integration from various sources into centralized repositories.Built and maintained data mo
Designing building and optimizing data pipelines for efficient data processing and integration.Migrated 5TB+ of data across platforms ensuring seamless integration and performance.Performed data migration and integration from various sources into centralized repositories.Built and maintained data models and schemas to support business intelligence and reporting.Ensured data security and compliance with industry standards and regulations.Develop Extract Transform Load (ETL) processes to cleanse enrich and transform data for downstreamconsumption.
Show More

Data Engineer II

2021/09 - 2022/06

Skills:

Your Role and Responsibilities:

Build and maintain scalable data pipelines from various data sources into multiple data warehouses.Migrated data from data lakes data warehouses and data marts.Collaborate with cross-functional teams to understand data requirements and deliver solutions that meet businessneeds.
Build and maintain scalable data pipelines from various data sources into multiple data warehouses.Migrated data from data lakes data warehouses and data marts.Collaborate with cross-functional teams to understand data requirements and deliver solutions that meet businessneeds.

Education

2015 - 2017

2009 - 2013

2007 - 2009

2006 - 2007

Skills

Amazon Relational Database Service Apache Spark API Development Artificial Intelligence Big Data
+ More

Hobbies

Traveling and learning new things. 

Achievements

  1. Spot Award  - Bristlecone
  2. ‘Remote & Stellar - A Beautiful Mind’ - Bristlecone
  3. Spot Award  -  Bristlecone
  4. Spot Award  -  Bristlecone
  5. Best team player - Autodesk

Certification

Data science for engineers, IIT madras

Preferred Languages

English - Fluent Hindi - Fluent