Created a big data pipeline using Kafka for a stream pipeline

And a batch pipeline using spark and sqoop and using big query as database

All of above was done on Google cloud platform

Understand the requirement and creating a pipeline for ETL based on client use case.  Working on a framework for bulk ingestion which also handled multiple scenarios like PII tables handling ,ingestion of tables with or without specific column etc.   Automation of the whole process of ingestion and validation of data according to client requirment.   Worked on data transformation and data handling of ingested data into the raw layer.