Gathering requirements, mapping data, designing project architecture, database transitions, migration of application code to Spark, shifting applications to Kafka streaming data, team management, AWS services integration.

Analyzing payment notifications, creating alerts, analyzing amounts for recurring transactions, data ETL application using Scala Spark, SQL, and Hive queries, Apache Airflow for orchestration.

Migrating data from Oracle DB with AWS DMS, processing and storing streaming Data into AWS Aurora Postgres DB, EMR for data transformation, PySpark for dataset processing, AWS step functions for batch data processing, IAAC AWS CloudFormation, Glue Data catalogue configuration, AWS Athena queries audit, Glue ETL for ad hoc data processing.

Processing Data stored in ADLS using Azure Databricks, CI/CD implementation using Azure DevOps, PySpark job orchestration using Azure Datafactory, Optimization of spark jobs.

1.Dealing with huge learning data of entire organization.

2.ETL operations on the data.

3.Generating reports and dashboards with this data.

Design, creation of datasets using Hadoop/Hive, Apache Spark framework, Hive querying, Shell scripting, Azure-DevOps pipelines, GIT, Kafka-connect, Cloud Migration to Google Cloud Platform, ETL on Google Cloud Platform using BigQuery and Dataproc, Big Query analytics, DataProc data processing, Google cloud Composer (Airflow) orchestration, Terraform with Azure DevOps.