Senior/Lead Big Data Developer

Position:

• Designing Hive/HCatalog data model includes creating table definitions, file formats, compression techniques for Structured & Semi-structured data processing
• Implementing Spark processing based ETL frameworks
• Implementing Big data pipeline for Data Ingestion, Storage, Processing & Consumption
• Modifying the Informatica-Teradata & Unix based data pipeline
• Enhancing the Talend-Hive/Spark & Unix based data pipelines
• Develop and Deploy Scala/Python based Spark Jobs for ETL processing
• Strong SQL & DWH concepts

• Function as integrator between business needs and technology solutions, helping to create technology solutions to meet clients’ business needs
• Lead project efforts in defining scope, planning, executing, and reporting to stakeholders on strategic initiatives
• Understanding of EDW system of business and creating High level design document and low level implementation document
• Understanding of Big Data Lake system of business and creating High level design document and low level implementation document
• Designing Big data pipeline for Data Ingestion, Storage, Processing & Consumption
• Must have strong Big Data, Hadoop, Hive, HDFS, Spark, YARN
• Strong understanding and knowledge in Data Warehouse concepts
• Experience in SQL
• Expertise in Linux/Unix (OS)
• Must have strong Programming Language experience in Scala and/or Python

• Informatica
• Talend
• Java
• Teradata
• MySQL
• Oracle