Data Engineer AWS

Topic Expectations

SQL

• Proficient and hands on with the following -
• Database normalization, ER modeling
• Working with Databases and tables
• Functions in SQL
• Subqueries, Operators and Derived
• Tables in SQL
• Window functions in SQL

CTEs

• Working with Views
• Stored procedures and triggers in SQL
• Performance Optimization and Best
• Practices in SQL

Python

• Data Operations
• Conditional Statements and
• Functions
• Error Handling and File Operations
• Shell Scripting
• Pandas

AWS

• Collection
• Storage (S3)
• Data Processing
• ETL with Redshift,Glue
• Resilient Distributed Datasets
• DataFrames and Transformations
• Data Processing with Spark, Pyspark
• Security
• Reading and Writing csv, json, parquet file format data from and to AWS S3
• Analytical queries with windowing functions against S3(functionalities like lead, lag, Partition By...)
• Preparing datasets from a big data environment(S3) using complex joins
• Deduplicating datasets ( json, parquet, csv, SQL formats )
• AWS Glue basics - DataCatalog to discover metadata, Glue jobs for ETL
• AWS Lambda
• Compression concepts. Parameterization.
• Event handling. If the subscriber is down, what happens to the sent events, how would you handle it. If the function fails on handling the event, what happens? How do you handle it?
• Debugging and RCA (Root Cause Analysis)
• Error/exception handling and monitoring
• Scheduling jobs