About Me
Hello I am a Big Data Enginner with 3 years of exp. I have a excellent Skills in Hadoop,Spark,Scala,Python,Hive,Hbase,Pig,MongoDB,Kafka,SparkStreaming,Apache Airflow,Dataiku, Apache NiFi and so many Big Data Tools which is required
... Show MoreSkills
Others
Web Development
Database
Programming Language
Software Engineering
Development Tools
Portfolio Projects
Company
Working With Microsoft Server To Spark TO ELK process
Description
We are getting Data From Microsoft SQl server to Spark Perform a Data Transformation and AI/ML On that Data and after that stored That Data into ElasticSearch and that JSON formated Data used into Kibana's BI Report
Show More Show LessSkills
Apache Spark Spark SQL SQL Kibana ElasticSearchTools
IntelliJ IDEA Postman KibanaCompany
Kafka Spark Streaming With Kinesis and ELK
Description
Data is getting from the senser using Iot Device and with using MQTT Broker we get that Data Into Spark with Using Spark Streaming or else we get that Data from Kafka Spark Streaming it is continuesly getting Data From Kafka and We perform Data Transformation on Spark and After that we have to stored that Data into ElasticSearch and kinesis and that Data is used From Front End side as a JSON Token as per there requirements
Show More Show LessSkills
Apache Spark PySpark Scala ElasticSearch Apache-Kafka Spark Streaming Message Queuing Telemetry Transport - MQTT Kinesis KibanaTools
IntelliJ IDEA PostmanCompany
Data warehouse Hive to Data ingestion with Sqoop with Spark and Stored into HDFS
Description
Data is available in Hive and I need to used that Data With using Different File formats With using Sqoop.
In Sqoop it is a used For Data Ingestion Tool which processed and transfer Data From Hive to HDFS as per Client's requirements.
If they Required Something with using Spark then used Spark SQL directly
Show More Show LessTools
IntelliJ IDEA Terminal Hive