Senior Data Engineer

job summary:

• Lead, design and implement features in collaboration with team engineers, product owners, data analysts, and business partners using Agile / Scrum methodology
• Contribute to overall architecture, frameworks and patterns for processing and storing large data volumes
• Design and implement distributed data processing pipelines using Spark, Hive, Sqoop, Python, and other tools and languages prevalent in the EMR ecosystem
• Build utilities, user defined functions, and frameworks to better enable data flow patterns
• Research, evaluate and utilize new technologies/tools/frameworks centered around high-volume data processing
• Define and apply appropriate data acquisition and consumption strategies for given technical scenarios
• Build, incorporate, and drive teams to write automated unit tests and integration scripts using tools like the Python test unit framework
• Work with architecture/engineering leads and other teams to ensure quality solutions are implemented, and engineering best practices are defined and adhered to
• Work across teams to resolve operational and performance issues and to deliver data for reporting in Tableau
• Implement security control around sensitive data

• Experience level: Experienced
• Minimum 5 years of experience
• Education: Bachelors

• ETL
• Python
• Snowflake
• APIs
• AWS
• Hadoop