Description

Client is seeking a Senior Data Engineering to assist in launching two cutting edge iOT advanced analytics platforms currently rolling out to internal and external partners. The Senior Data Engineer will be responsible for the development, maintenance and optimization of data migration and CI/CD pipelines across the full scope and stack of our platforms. This role has a strong understanding of SDLC principles as they apply to DevOps and Data Engineering including design, development, maintenance and deployment of scalable Azure cloud data, CI and CD pipleines.

Duties & Responsibilities

• Write and maintain reusable DataFactory pipelines for automating data transfer from on premise legacy systems into Azure cloud.
• Collaborate with team members to support and improve data pipelines, data migration processes, release pipelines, standards and tools.
• Maintain and troubleshoot existing Azure DevOps pipelines for application deployments.
• Use best practice to engineer re-usable configuration based code and pipelines.
• Solution and implement best practice for mapping legacy relational schemas to Azure SQL and Azure table storage components.
• Implement, and test automated data migration, backup and recovery procedures across multiple cloud and on premise systems.
• Analyze, diagnose and repair failing pipelines.
• Write and maintain clear, concise documentation, runbooks and operational standards including data flow and ORM diagrams.

Qualifications

• 4-year bachelors degree in Computer Science, Computer Engineering or related discipline
• 7 years of data engineering large scale, low latency relational data platforms.
• 3 years experience building data pipelines with Azure DataFactory.
• 3 years experience managing build and release pipelines with Azure Functions, containers, messaging, and orchestration components.
• 4 years experience deploying solutions on Azure cloud adhering to Microsoft Azure .
• 4 years experience working within Agile development organizations.
• Extensive ETL/ELT and data modeling experience.
• Extensive hands-on experience optimizing and building data pipelines leveraging Microsoft SQL
• Extensive hands on experience developing big data transformation pipelines mapping various data structures to relational optimized relational structures.
• Extensive experience with version control(git) in a CI/CD development workflow.
• Working knowledge with modern DevOps solution patterns and Infrastructure as code cloud automation solutions.
• Strong OOP principles and practice with Python, Java, or C#.
• Experience building data pipelines to support batch, microbatch, streaming, messaging and REST based API integrations.
• Experience with Azure DevOps application release pipeline management.
• Very strong verbal communications skills; comfortable communicating with technology leadership.