Creating Real time and distributed pipeline for Sleepiz India Using Cassandra as Database dump. Fetching real-time data from devices generating values from electromagnetic waves and processing it further for Digital signal processing and Machine learning programs. Fetching data for UI presentation to customers who are doctors.

Created a big data pipeline using Kafka for a stream pipeline

And a batch pipeline using spark and sqoop and using big query as database

All of above was done on Google cloud platform

Understand the requirement and creating a pipeline for ETL based on client use case.  Working on a framework for bulk ingestion which also handled multiple scenarios like PII tables handling ,ingestion of tables with or without specific column etc.   Automation of the whole process of ingestion and validation of data according to client requirment.   Worked on data transformation and data handling of ingested data into the raw layer.