Delivered an efficient,smooth pipeline to migrate data from sql server to snowflake using data lake as azure blob storage with the help of azure data factory and developed a alert system to state if the pipeline fails over platforms like email and Microsoft teams using logic apps.

Here the role needed to fetch the data from outlook emails in pdf format and preprocess it and upload the processed data to snowflake .

Here i had a minor role to tranfer the data from cli instance to snowflake. The challenge over here the team was facing was to uploading the data with more than 6 million records to snowflake using kafka . Here i wrote a script in python to upload the data directly to snowflake in the cli instance