JOB DESCRIPTION:

• Experience writing, configuring, and running Spark applications and use of SparkSQL a MUST
• Experience with Test Driven Development and DevOps
• Experience with using GitLab for code management
• Development experience in data warehouse or data lake environment
• Understanding of change data capture, incremental loads, and other techniques used in data pipelines
• Understanding of data design (3nf and dimensional)
• Familiar with full-stack development practices
• Understanding of Functional Programing practices
• Work with the DE’s and Product owners to understand the requirements & Design.
• Develop code and perform unit testing in Hadoop with Pyspark
• Perform peer review
• Promote the code to QA and prepare production migration checklist
• Provide SIT, UAT, Go-Live and post implementation support
• Responsible for working with DE’s to resolve technical issues/challenges to keep project on track
• Work with the DE’s and Product owners to obtain the required approvals