About Me

Having 10 years of IT experience in Development & Enhancement of Data warehouses and have worked on Big Data applications using various tools and technologies like HDFS, Hive, Sqoop, Impala, Python, PySpark, Snowflake, Databricks, Informatica, Terada...nologies like HDFS, Hive, Sqoop, Impala, Python, PySpark, Snowflake, Databricks, Informatica, Teradata and Unix Shell Scripting and good experience in AWS cloud services. Having sound knowledge in Data warehousing/Big Data Architecture, Data Lake and Technologies. Good working exposure on cloud technologies AWS- EC2, S3, Lambda, Glue, Athena, SQS, SNS. Designing and implementing data ingestion pipelines from multiple sources using Apache Spark and/or Databricks. Integrating the end-to-end data pipeline to take data from source systems to target data repositories ensuring the quality and consistency of data is maintained at all times. Involved in migration of data from Teradata to Snowflake. Hands on experience in loading the data to Snowflake Data warehouse and managing the data in snowflake. Created Snowflake external tables, stages, snow pipe, streams, views, Snowpark and Procedures. Hands on experience on major components in Hadoop Ecosystem like HDFS, Hive, Impala, Sqoop, PySpark and Parquet. Ingest the salesforce data to data lake using Streamsets community/enterprise editions. Experience in analyzing data using Hive QL and creation of impala views for business users. Experience in importing and exporting data using Sqoop from RDBMS to HDFS and vice-versa. Pre-processing of HDFS data using Hive and PySpark. Good at applying optimizing methods to avoid the memory issues and improve the spark performance. Experience on loading and manipulating large datasets using Spark SQL. Knowledge on CI/CD tools like Jenkins, Artifactory, bit bucket, Jira and Ansible tower. Knowledge on Terraform/Cloud formation templates to provision the AWS services. Good exposure in Data Analysis, Data Cleansing, and Transformation, Integration, Data import, Data export and use of ETL tool including Informatica. Experience on scheduling workflows through Airflow, Oozie, Autosys, UC4, Control-M. Worked on different domains like Health care, Retail, Banking and Insurance.