Involved in development of spark/spark sql query to run in prod 

Roles and Responsibilities:

·        Implemented newer concepts use like Apache Spark and Scala programming

·        Managing data coming  from 200+ different sources

·        Loaded unstructured data into Hadoop File System( HDFS)

·        Written validation and data quality scripts

·        Implementation of Cloudera cluster with High availability and standby solutions.

·        Design and support of Data ingestion, Data Migration and Data processing for BI and Data Analytics.

·        Responsible for developing data pipeline Sqoop and pig to extract the data from weblogs and store in HDFS.

·        Involved in developing Pig Scripts for change data capture and delta record processing between newly arrived data and already existing data in HDFS.

·        Worked on HIVE- Integration-Spark SQL scripts for performance enhancement

·        Worked on DataFrame Development as part of SparkSQL.