Now you can Instantly Chat with Dinesh!
- Overall 6.6+ Years of IT experience in Software development Big Data, Hadoop and Spark Technologies.
- Hands on Experience in Hadoop Technologies like HDFS, Hive, Sqoop, Apache ...
- Involved in capability building for developing real time streaming applications using Apache Spark.
- Performance tuning of Spark, use file format like(Avro, Orc, Parquet)
- Hands on Experience in Data Modelling in Hive.
- Experience in CDH and Hortonworks environment.
- Experience in Build the Nifi job
- Sound knowledge of Hadoop commands.
- Knowledge in Azure Blob Storage, Cloud Engineering with GCP, Amazon services.
- Strong Knowledge of Agile (Scrum) methodology.
- Domain Experience in Pharmaceutical, Retails, Banking.
- Involved in release, Go Live activities.
- Good communication, interpersonal, analytical skills, and strong ability to perform as part of team.
- Willing to update my knowledge and learn new skills according to Business Requirements
Data & Analytics
Technology: Hadoop, Hive, Kafka, apache Nifi, HDP.
Description: the project which will deliver a pilot model for a centralised data storage and
data management platform. This will be used to store and process the data from the log files
that come from various IT and Network applications into ArcSight.
- Build the nifi job for Kafka topic to hdfs (raw zone)
- Build the stream flow for the serving layer(stream data).
- Develop DDL for store the data into hive table
- Implemented performance tuning.
Enterprise Data Hub(EDH)
Technology: Hadoop, Sqoop, Hive, maven, shell script, Spark, Spark Streaming, Scala, Kafka.
Description: This Project is responsible for Developing Data lake to analyse customer data of a leading bank which allows client to have clear picture of customers and their activities. Project has two phases as Ingestion and Data Manipulation. First phase of this project is to develop a Ingest mechanism to pull data from source systems like DB2 and CDC to HDFS platform.
- Develop automation for Sqoop job for multiple table with provided mapper.
- Develop spark code for moving data from staging layer to schema, raw and archive layer with add additional audit column
- Develop spark code for collecting data from Kafka topic to staging layer(HDFS).
- Execute all script using batch scripting.
Toolshadoop Hive Spark Streaming sqoop
Technology: Hadoop, S3, EMR, EC2 ,AWS, Spark core and python.
Description: Adtech ingestion is a ingestion the data from multiple source location to data lake and the date lake is on S3 premises for storage. The data manipulation and cleansing perform using spark and python. Validate the data in data warehouse using Hive and redshift.
- Involved in manage data coming from different DB to S3 using spark python.
- Writing CLI commands using HDFS and S3.
- Involved in creating Hive tables, loading with data which will run internally in Map Reduce way.
- Implemented complex Hive and red shift queries for validate data.