Problem statement: Dev team need to identify and calculate the performance after optimizing/refactoring the code of the crawler.

My task here is to calculate the performance of the crawlers manually by executing crawlers on various thresholds.

KILN is a versioning system like GitHub.

Problem Statement: Client wants us to record a script that performs all actions that a versioning system can perform. i.e. Creation of repo, checkin, checkout, modifications, review, raising PR, push, pull etc. And during the execution he wants to see how system behaves while we fluctuate a load onto the system. We have a defined time for putting the stress andload on the system and all need to verify the performance of the system during that time.

These are few projects among 150+ projects that we monitor on weekly basis. We have a release cycle SE7 (release every 7days), to release the product we have maintained some quality bars around the executions.

For this we created a framework using Jenkins, Jmeter, Influx DB, Grafana, and New relic to monitor the performance impact of various changes within the projects.

FirstRain is a product and services based company. It’s product named as Market intelligence, is a very big web application constructed on JAVA and Solr platform. The end result of the product is to make available different type of documents that are fetched from different web locations which are helpful to make their managerial decisions. We have all our component deployed and managed at our end.

But the problem scenario here is that we have to build a product around an input stream that we are unable to view, we cannot bring the test data at our dev environment to view what we have to build and will our solution help the client in a well-defined manner. We have design, create, test and deploy a solution to their problem on our imaginations within their own network and Firewall in a limited set of resources. The best part is that not only we achieve the solution to their problem but we achieved a new solution towards filtering of content that is never seen in my company’s history and we are trying to implement the same in some more internal components in FirstRrain.

My current KRA. It started with the upgrading old C, C++ based crawler to java crawler. The whole framework is based on scheduling of crawling different source sites. Crawling Schedule of different site is based on several parameters, like robot.txt of the site, crawl frequency, identification of bad classes on the site that holds either advertisements or non-usable data, identification of old documents, crawling secured sites, etc. This is an on-going process, as soon as we find glitches on the inflow of the documents we work on the resolution of that glitches. We have different automation process that helps us to identify the extraction issues, like document sniffers, source sniffers. Currently we are working on crawling of javascript pages of a website, using selenium.

This is a huge architectural change in product. In brief we decided to break the big data into small chunks that should be easily manageable. We broke the FirstRain Product into two parts: Entity and Docs. Basically we defined entity as a rule that controls the inflow of the documents. Either it is source definition or entity definition all should be brought at a place and there should be an application that should give analytics team, an interface from where they can visualize and modify the entity data. Henceforth we come across an application named Entity workbench. Now we have a monitoring application over the entities, but we need to increase our entity base. That requires a concept to add entities and there metadata in bulk. For this we integrated the system with a third party company feed that allows us to provide the basic metadata of the companies in format, which can be easily plugged into the system. Along with this it helped to keep the system up-to-date. As a proof of concept we are now managing to serve 2.5 million entities in an automated/manual manner to the clients in 2016. [Whereas, we targeted 0.2 million entities in 2013]. This whole process plays a vital role in the companys success.

This was a challenge. I have to work on a framework whose responsibility is to verify the correctness of the feed that is the response of the requests sent by the mobile APP installed on different mobile devices (Android, iOS, Windows). For this I designed and developed a framework that takes input from the application logs of each day (real time activities that are done by the actual users on mobile app) and verify the response s and its response data. This is done through different algorithms written in the excel file and compares the actual results with the computed results.

Aurea is company holding all the development responsibility of ESW Capital. Joined this company as a QA Engineer as a part of company acquisition. The company works on the concept of Virtual office, where we have to collaborate and co-ordinate people across the globe. For the initial 4 months they asked me to work on many stream: UI Automation (Cucumber and selenium framework) API Automation (REST assured framework) Test case writing (Test Rail) Framework enhancements From Feb. 2018, I was asked to channelize a new performance testing team and represent a POC and values of the new stream. The result was very productive, and we are able to represent the importance of performance testing to the higher management. From that time, we are trying to produce a quality output in-time. Now, being a team of 22 people, me (as an Architect) and an acting people Manager are responsible for managing ~130 odd performance testing suites running nightly and well monitored using Jenkins and GRAFANA dashboards. In past one and a half year we have successfully achieved following: Provided a new concept of performing Client side performance testing using JMeter, Selenium and Selenium Grid. Automating a functional testing workflow of product named as KILN, which is a Versioning system like Github. Analyzing the product behavior when there is a Hoax attack on the system. And many more.

FirstRain Inc. was a Product based Service provider company. The product (Market Intelligence) provides an analytics platform that brings together external and internal unstructured data to coach teams and professionals on key business developments that affect their go-to-market strategy, business relationships, and investments.

At this stage Market Intelligence was evolved at a level, that we molded the system in such a manner that it should be able to provide analytics, management changes, and all the business relevant documents that are available on the open web to the user as soon as he performs search into our system. From here we started using SOLR as the single backend input media for all the frontend components. I.e. either they are feeds, emails or the searches that are been executed by the system uses SOLR as a single point of contact where all the data is available. This step boosted the systems performance in an exponential manner.