Job Description

• Understand the design specifications.
• Follow the best practices for core product development.
• ​Data ingestion for distributed system with parallel processing, in Java, CPP or Golang for  
• from cloud storage – Amazon S3/Azure Cloud Storage/Google Cloud Storage, relational database – Snowflake, Google BigQuery, PostgreSQL, Files, Kafka, data lakehouse (Iceberg) 
• High-availability (HA) loading, cross-region replication 
• Loading monitoring/error reporting  
• Spark Connector  
• Third-party management – Kafka, Kafka Connect 

Requirements

• building distributed systems applying applying parallel processing.
• Any or of the following – Kafka, Zookeeper, Spark, Stream processing.
•  Experience in Kafka Connect, Kafka stream, Kafka security and customization
•  Spark connectors
• Event driven architectures.
• Agile development 
• CICD 

​

Nice to have:

·       gRPC protocol experience and multi-threading

·       Zookeeper/ ETCD/ Consul

·       Familiarity with distributed consensus algorithms (Paxos / Raft)

·       Docker, Kubernetes

​