Working on processing of real data json data stream using Spark and Kafka Integration. We store intermediate data in redis and final output we were stored in Mongo DB.

And We updated data in Mongo DB using real data stream.

Open Item batch is an item information maintenance system. It holds product information. An item identifies the Division, Category, Store location and Selling price (Retail) of the merchandise. It assigns the Key Stock Number (KSN) which signifies the product purchased by the customer. KSN identifies the style, size, color and flavor. Open Item batch is not an ordering system.

Data for tables is extracted from Teradata. This data is used by other systems such as Local Publishing or Dynamic Pricing. Item and Pricing information must be extracted from Enterprise Data Hub (HADOOP)

The main purpose of this project is to process data coming from multiple sources and apply transformation and store the data in Hive. We are developing data pipelines and schedule them on Kylo.

Technology Used - Spark, Scala, Nifi, Kylo, Azure

Processed data coming from multiple sources and apply transformation and store the data in Hive. We are developing data pipelines and schedule them on Kylo.