Big Data Lead | Online Jobs | Optimhire

Big Data Lead

What will you do?

  • To help us go to the next level we are looking to onboard a hands-on SME in leveraging big data tech to solve the most complex data issues. You will spend almost half of time with hands-on coding.
  • It involves large scale text data processing, event driven data pipelines, in-memory computations, optimization considering CPU core to network IO to disk IO.
  • You will be using cloud native services in AWS and GCP.

Who Are You? 

  • Solid grounding in computer engineering, Unix, data structures and algorithms would enable you to meet this challenge.
  • Designed and built multiple big data modules and data pipelines to process large volume. 
  • Genuinely excited about technology and worked on projects from scratch. 

Must have:

  • 7+ years of hands-on experience in Software Development with a focus on big data and large data pipelines.
  • Minimum 3 years of experience to build services and pipelines using Python.
  • Expertise with a variety of data processing systems, including streaming, event, and batch (Spark, Hadoop/MapReduce)
  • Understanding of at least one NoSQL stores like MongoDB, Elasticsearch, HBase
  • Understanding of how data models, sharding and data location strategies for distributed data stores in large scale high-throughput and high-availability environments and their effect in non-structured text data processing
  • Experience with running scalable & high available systems with AWS or GCP.

Good to have:

  • Experience with Docker / Kubernetes
  • Exposure with CI/CD
  • Knowledge of Crawling/Scraping

Job Type



Full-Stack Developers

Must have Skills

  • Python - 3 Years


  • MongoDB - 1 Years


  • Big Data - 3 Years


  • Cloud Computing - 2 Years



english - Fluent

21 - 48 K/Year USD (Annual salary)

Longterm (Duration)

Fully Remote

Bhavesh P