Job Description

Our client is looking for a Data Engineer/Scientist (Advanced) with Bachelor's or Master's degree in Computer Science, Software Engineering. minimum of 3 years' experience as a Data Engineer, 2 years of experience working with AWS services must be based in Gauteng, contract December 2027
Essential skills requirements:

• Proficiency in Python 3.x for data processing and automation.
• Experience with AWS Glue for ETL processes.
• Strong knowledge of AWS Athena for querying large datasets.
• Hands-on experience with AWS Lambda for serverless computing.
• Familiarity with AWS EC2 for scalable computing resources.
• Expertise in AWS CloudWatch for monitoring and logging.
• Proficiency in working with PostgreSQL RDS for database management.
• Experience with AWS QuickSight for data visualization and reporting.
• Strong understanding of data ingestion pipelines, particularly for Call Detail Records (CDRs).
• Proficiency in Git and GitHub for version control and collaboration, including experience with GitHub CI/CD pipelines.
• Any additional responsibilities assigned in the Agile Working Model (AWM) Charter

Advantageous skills requirements:

• Extensive experience with Terraform for infrastructure as code, including creating, managing, and optimizing AWS resources.
• Experience with other AWS services such as S3, Kinesis, and DynamoDB.
• Knowledge of data formats such as Parquet, AVRO, JSON, and CSV.
• Experience with Docker for containerization.
• Understanding of Big Data technologies and frameworks.
• Familiarity with Agile working models and tools like JIRA and Confluence.
• Experience with data quality tools such as Great Expectations.
• Knowledge of REST API development and integration.
• Strong analytical skills for troubleshooting and optimizing data pipelines.
• Experience in developing technical documentation and artefacts.

What will be your role and responsibilities?

• Design, develop, and maintain data ingestion pipelines for BMW's connected fleet, focusing on Call Detail Records (CDRs).
• Utilize AWS Glue for ETL processes to transform and load data into the analytics platform.
• Implement efficient querying solutions using AWS Athena.
• Develop serverless applications and workflows using AWS Lambda.
• Monitor and optimize data pipelines using AWS CloudWatch.
• Manage and maintain PostgreSQL RDS databases for data storage and retrieval.
• Create interactive dashboards and reports using AWS QuickSight.
• Leverage Terraform extensively to define, deploy, and manage AWS infrastructure as code, ensuring scalability and reliability.
• Use Git and GitHub for version control, collaboration, and maintaining code repositories.
• Implement and manage GitHub CI/CD pipelines to automate testing, deployment, and delivery processes.
• Ensure data security and compliance with BMW's information classification requirements.
• Stay updated with the latest data engineering tools, technologies, and industry trends.
• Identify opportunities for process improvements and automation to enhance pipeline efficiency.
• Collaborate with cross-functional teams to understand data requirements and deliver solutions.
• Develop and maintain technical documentation for data engineering processes and solutions.

Apply now!!! send CV to
For more information please contact:
Natally Tshabalala

OpenSource Intelligent Solutions