Job Description

Position summary
Industry: FMCG & Supply Management
Job category: FMCG, Retail, Wholesale and Supply Chain
Location: Cape Town
Contract: Permanent
Remuneration: Market Related
EE position: No
Introduction
Data Engineer who will assist in designing and implementing scalable and robust processes to support the data engineering capability. This role will be responsible for extracting and transforming massive amounts of data at scale and consolidating this data into a bigger data ecosystem.
Job description

• Design, build, and maintain scalable, automated data pipelines using AWS services like Glue, Lambda, Step Functions, EMR, and Kinesis.
• Develop ETL/ELT processes to ingest, transform, and deliver structured and unstructured data into data lakes, data warehouses, and analytical environments.
• Ensure efficient batch and real-time data processing, optimising workflows for performance, scalability, and cost efficiency.
• Implement robust data movement strategies, ensuring secure and efficient data transfer across cloud storage layers.
• Develop and maintain data transformation logic, ensuring data is structured for engineering, AI/ML, and analytics use cases.
• Implement change data capture (CDC) mechanisms to support real-time data synchronisation and streaming analytics.
• Optimise large-scale data processing workflows for performance and efficiency.
• Ensure data pipelines feed into AWS-based data lakes (S3), data warehouses (Redshift), and feature stores for AI/ML.
• Enable AI/ML and analytics teams with structured, high-quality data, ensuring seamless integration with their models and insights.
• Collaborate with engineering teams to support automated, self-service data access.
• Develop CI/CD pipelines to automate testing, deployment, and monitoring of data workflows.
• Leverage Infrastructure as Code (IaC) tools like Terraform and CloudFormation for scalable cloud data infrastructure automation.
• Implement version control, rollback strategies, and automated alerting for pipeline reliability.
• Continuously optimise data pipelines to improve performance, reduce latency, and minimise operational costs.
• Stay up to date with advancements in cloud-based data engineering and best practices.

• Bachelor's degree in Computer Science, Business Informatics, Mathematics, Statistics, or Engineering.
• 4 - 5 years of relevant data engineering experience in cloud-based environments.
• AWS Certifications (AWS Data Analytics, AWS Solutions Architect) are advantageous.
• Strong understanding of data structures, algorithms, and software design applied to cloud-based data engineering.
• Experience working with structured and unstructured data at scale, including key-value, document, and columnar data stores.
• Proven expertise in AWS services, including Redshift, Glue, Lambda, EMR, S3, IAM, RDS, and Athena.
• Experience building and optimising cloud-based ETL/ELT pipelines for analytics, reporting, and AI/ML workloads.
• Strong programming, performance tuning, and troubleshooting skills using Python, Scala, Java, and C .
• Hands-on experience with distributed data processing frameworks, including Apache Spark and AWS Glue.
• Experience designing and implementing AWS-based data solutions, including data lakes, event-driven architectures, and API integrations.
• Experience working with streaming data solutions, including Kafka, Kinesis, and change data capture (CDC) synchronisation.
• Proficiency with DevOps-oriented data engineering, including CI/CD automation and Infrastructure as Code (Terraform, CloudFormation).
• Experience with version control systems, including Git and SVN.
• Expertise in monitoring and troubleshooting data pipelines using AWS CloudWatch.

• Analytical Mindset: Demonstrates a strong analytical and problem-solving ability, capable of breaking down complex data issues and devising effective solutions.
• Collaboration and Communication: Exhibits excellent interpersonal and communication skills, with the ability to articulate complex data concepts to non-technical stakeholders. Must foster a collaborative team environment and efficiently work across different departments.
• Continuous Learning: Has a strong commitment to continuous professional development, staying ahead of the latest trends and technologies in data engineering and analytics. Willingness to pursue relevant certifications and training.
• Innovative Thinking: Displays innovative thinking and a proactive approach to identifying and pursuing opportunities to improve data processes and solutions. Comfortable proposing and experimenting with new technologies or methodologies to enhance data capabilities.
• Adaptability: Demonstrates flexibility in adapting to changing business needs and technology landscapes. Can efficiently manage multiple priorities and adapt strategies in a fast-paced environment.
• Cultural Fit: Aligns with the organisation's culture and values, contributing positively to team dynamics and company morale. Demonstrates integrity, accountability, and a strong work ethic.