This is regarding the design of data pipeline for the car data for different parts which are used to compare with the original data to identify the defects present in the vehicle. The data will be prepared and will be deployed in AWS cluster.

This project is regarding streaming the application data from HDFS and send to the end user directory for 8 application feeds. The Spark Streaming application is developed using Scala in a Generic Extraction Framework (GEF) model, which processes data by referring the parameters in property file.

This project is regarding generation of Customer Score reports at different granularities in Hive based on the customer data. The customer report generated is integrated with Tableau to visualize the information generated in the system. The logic present in Hive is being migrated to Pyspark in order to reduce the execution time of the entire process and also to handle more complex data faster.