• Created Big Data Pipeline and Implemented Spark based Model for batch processing of data.
• End to end product features development.
• Ingesting data from sftp server to snowflake via aws s3 using JDBC connection
• Ingesting data from sftp server to postgress via snowflake using REST API
• Trained interns for Big data Technologies.
• Developed pyspark code to validate data and load data to snowflake from s3 Ingesting data from Azure blob and loading the data to Postgress . Schedule the entire flow using Apache Airflow

• Ingest data from sql database to hive with business transformation.
• Script creation to generate reports for BI team from Hive using HIVEQL.
• Contributed in enhancements and performance improvement of the process.
• Involved in performance improvement activity in Hive with Joins,Group and aggregation
• Created automation process to schedule all scripts using Oozie/Falcon.

PNDA is an Open source Platform for Network Data Analytics.Efficiently distributes data with publish and subscribe model.Processes bulk datain batches,or streaming data in real-time.Worked as a coredeveloper of PNDA.Have added new features,fixed manyi ssues.Developed Kafka custom producers, developed some batch and stream spark applications with python for PNDA