Project Description: BigData Development Process huge amount of data as part of day to day operations. Worked with Claims Assure & Corporate Records team to understand the underlying business logic and loaded those bulk data for analysis. By data ingestion, maintained huge data and performed data transformation/cleaning, developed predictive data models for business users as per requirement. • Developed complete end to end Big-data processing in hadoop eco system. • Optimized Hive 2.1.1 scripts to use HDFS efficiently by using various compression mechanisms. • Used Spark API 2.1.0 over Cloudera to perform analytics on data in Hive tables. • Developed Oozie workflow jobs to execute hive 2.1.1, sqoop 1.4.7 • Created hive schemas using performance techniques like partitioning and bucketing. • Used Sqoop 1.4.7 to load Oracle tables in HIVE • Used Pig 0.17.0 for transformation of data in HIVE tables. • Involved in complete end to end code deployment process in Production. • Used SFTP to transfer and receive the files from various upstream and downstream systems. • Worked in exporting data from Hive 2.1.1 tables into Oracle database. • Involved in complete end to end code deployment process in Production. • Involved in some product business and functional requirement through gathering team, updated the user comments in JIRA.

ETL processing through Python Hive Spark Hadoop to transform load data from various data sources to target for data science team to consume.

Sourcing data from various data sources and loading to different targets.

Consolidate data from different infra sources to prepare reports for resource utilisation.