The main purpose of developing the project is to generate useful data that can be consumed for Tableau reporting. The report which need to be generate are from different subject areas like Surgery, Emergency Department, Infection Control, Oncology etc which deals with huge amount of data. In order to reduce the processing time at tableau end, we are developing an ETL project from scratch where we are taking structured/ semi-structured data either from Oracle Db or HDFS where we are implementing MapReduce using framework called Apache crunch and flatting the source data with the help of Hadoop technologies and storing the data in Verica. This will help the tableau developer to generate various reports.

• Working on developing crunch transformation which have both structure and semi structure data(Oracle or HDFS) as a source and storing it to either intermediate HDFS or Vertica

• Involved in gathering business requirements, analyzing the project and creating design and architecture of the project.

• Involved in Design Reviews and Code Reviews.

• Coded and executed test clients for various features and

  implemented Junit to test the unit level functionalities.

• Played Role of virtual lead and Architect.

• Developed the entire project from scratch and deployed to Prod

  single handed

• Working on creation of Strom topology for each subject area.

• Creation of Spouts and Bolts for particular Storm Topology.

• Creation of HBase table which will be the end result of Bolt.

• Involved in Design Reviews and Code Reviews.

The main purpose of developing the project is to verify all the data are correctly processed and inserted into HBASE which will be used by the downstream project to process the ETL job. We are making use of the Apache Storm to push the data into HBase. We are creating two table namely Standardized and Normalized for each subject area (Encounter,finance,Person). The data we are processing is near Real-time which means each business day.