• Created data pipeline using Azure data factory
• Gathering the data stored in Azure data store, optimizing it and joining with internal datasets to gather meaningful information.
• Data cleaning using U-Sql code on the Azured data lake.
• Data aggregation on terabyte datasets done using U-Sql which saves IO/CPU time
• Adopted DAX for business logic calculations in providing analytics of system up time on periodical basis

Facilitated insightful daily analyses of 100GB to 1TB of server data collected by server logs. Spawning recommendations and tips that increased page performance by 38%.

• Developed spark scala programs to parse the raw data, populate staging tables and store the refined data in partitioned tables in the EDW.
• Created Hive queries that helped market analysts spot emerging trends by comparing fresh data with EDW reference tables and historical metrics.
• Enabled speedy reviews and first mover advantages by using Oozie to automate data loading into the Hadoop Distributed File System and PIG to pre-process the data.
• Provided design recommendations and thought leadership to sponsors/stakeholders that improved review processes and resolved technical problems.
• Managed and reviewed Hadoop log files.
• Tested raw data and executed performance scripts.
• Shared responsibility for administration of Hadoop, Hive and Pig

• • Created HBase tables to load large sets of structured, semi-structured and unstructured data coming from UNIX, NoSQL and a variety of portfolios.
  • Supported code/design analysis, strategy development and project planning.
  • Created reports for the BI team using Sqoop to export data into HDFS and Hive.
  • Developed multiple MapReduce jobs in Java (spark) for data cleaning and pre-processing.
  • Assisted with data capacity planning and node forecasting.
  • Collaborated with the infrastructure, network, database, application and BI teams to ensure data quality and availability.
  • Administrator for Pig, Hive and Hbase installing updates, patches and upgrades.

· Performed analysis of current optimization model and provided recommendations to move to Oracle’s cost-based optimization. Lead the stress test initiative to evaluate the success of this approach

· Established and wrote the Performance Tuning guidelines for the current module to enhance the quality of our product for all current and future code development

· Steered data mart and data warehouse development using 10.2, taking full advantage of new RDBMS features as they become available and stable

· Followed OMS standards (as per CMMI 1.2) in developing and building new applications