This is something which I had always wanted to built when started pursuing my engineering degree. This question had always chattered in my mind, how does big giants do the parsing and data sharing? Hence created to enhance my development|architectural skills. The complete architecture consists of individually scalable micro-services. Site’s basic components includes:
● Drupal 7 : Base for handling regular website functionalities ● Beanstalkd : Utilized by Drupal, for local jobs queuing ● CmdStalkd (GoLang broker for beanstalkd ) : Custom integration with Drupal, avoids long running PHP jobs aka listeners.
● ElasticSearch : For searching assets on the site.
● Falcon ( Python based web framework): Created API’s to process uploaded video/assets. Includes transcoding, overlaying of promo image, sprite images (scrubber/gallery), thumbnails, preview video, etc. All processing is handled using FFMpeg .
● Node.JS : Public container for making req/responses over web-sockets. Realtime stuff! ● Recommendations: Utilized by drupal’s computing module to execute a java based agent (open-source) which computes recommendation on a different cluster.
● Docker : Containerized Drupal, Falcon, ES, Node.js, Recommender services to scale independently.
● AWS : EC2, S3, CloudFront, RDS, Route53, EFS, etc. Right now there’s very minimal traffic and development is ongoing (since pet project so works only on weekends) so skipped ECS and used docker-compose on multiple EC2 instance in different AZs .
● Terraform : Complete setup is terraformed. Here’s an exmplet of ElasticSearch setup, terraformed & OpenSourced:
https://github.com/tkant/elasticsearch-terraform-ecs

Here we (team) had created a whole new micro-service called platform which resides on a server act as master and other slave servers hosts Cluster ( Platform Agent). The communication channel between these master and slave server is AMQP ( RabbitMQ ). This whole system is based on Request-Acknowledge fundamentals. Coded in core PHP which uses Kayako’s Novo Framework (Micro-service), it also includes Python bins on cluster, GoLang based beanstalkd broker, etc. Worked on GoLang based micro-service to read/write zones/nodes in Route53. Also modified a GoLang based broker called cmdstalk to enable processing of jobs on beanstalkd queue. Had also created an Elixir based broker to do the same stuff but management prefered the GoLang based one.
Small talk on ZooKeeper: https://prezi.com/p3fea51cnvs8/apache-zookeeper/ https://github.com/sensiolabs/consul-php-sdk : Added contribution to this repo.
Created shim classes in NOVO for integrating Consul in Kayako's infrastructure.
More brief details mentioned at:
http:// tusharkant.com/2016/09/huh-platform-engineer.html Technologies Used:
● Technologies Used: Custom PHP Framework (Novo), RabbitMQ, Beanstalkd, Python bins, GoLang (cmdstalk - beanstalkd broker), Redis, RESTful APIs (micro-services) , aws-cli, aws-sdk, Consul (had contributed to sensiolabs git repo), NGINX based load-balancing and hosts configuration automation, Dynect, DNS level round-level load-balancing, Chef etc.

Kanakuk Kamps - HT ( kanakuk,com ) : This whole site is solely developed by me! Current site was on Django so for better management client wanted to revamp the whole site in Drupal 7. Right now working over changes/modifications of the same. Technologies Used : Drupal 7, Solr Search, Varnish/Boost.

KSC-Kreate - RT (WallMart), an outsourced wallmart's project: An inventory management application used by "Wallmart". Technologies Used : CakePHP, ElasticSearch, Node.js, Redis.

Created RRI ’s first online PHD application form, launched at the end of every year. This whole process was done manually, till I introduced this concept and taken the confidence of stakeholders for building this brilliant site in Drupal 7.

Online wall decals store.

An inventory management application used by Wallmart.

An inventory management application used by Cost Plus World Market & The Limited.

This whole site is solely developed by me! Current site was on Django so for better management client wanted to revamp the whole site in Drupal 7. Right now working over changes/modifications of the same.

Porting current IMSC website from Drupal 6 to Drupal 7 on Debian 7 (wheezy) environment, using Drupal 7 debian package to make updates easier..

Created RRI s first online PHD application form, launched at the end of every year. This whole process was done manually, till I introduced this concept and taken the confidence of stakeholders for building this brilliant site in Drupal.

Complete setup is terraformed. Heres an example of ElasticSearch setup, terraformed & OpenSourced:

API written in Phoenix (Elixir based framework), kinda weekend project by Manager for me.

Rules engine in Rails, using Logic.