Job Description

Senior / Intermediate Data Engineer (Pinelands, Cape Town)

We are looking for a motivated and technically skilled Data Engineer who is eager to learn and grow. This role involves working with modern data platforms and development practices to design, build, and maintain scalable data solutions.

Key Requirements

Core Skills:

+ Strong communication and collaboration abilities. + Attention to detail with a focus on data quality.
+ Experience in end-to-end data flow design and implementation.
+ Familiarity with agile ways of working.
+ Growth Mindset. Demonstrated eagerness to learn and adapt, supported by examples of applying new skills in practice.

Technical Expertise:

+ Proven experience with big data tools, particularly Databricks. + Strong background in cluster computing using Apache Spark.
+ Experience developing and maintaining CI/CD pipelines.
+ Proficiency in Python with mandatory PySpark experience.

Beneficial Experience:

+ Prior work as a backend developer, showcasing strong software engineering fundamentals. + Previous experience as a data engineer.
+ Familiarity with Azure cloud infrastructure and services/

What You'll Do

Build and optimize data pipelines and workflows on Databricks and Spark. Contribute to the design and implementation end-to-end data flows and CI/CD processes. Collaborate with cross-functional teams to deliver reliable, scalable, and high-quality data solutions. Continuously expand your knowledge of modern data engineering practices.

What's in it for you?

Being South Africa's largest company of its kind - IDM is continuously growing, with no signs of stopping.


One of our core values is 'growing people' and we take that very seriously. We have internal learning opportunities to better our people, and leaders who are experts in their fields and freely impart knowledge and are open to new ideas. As one of our people said, "you can speak to anyone here. Your rank doesn't matter. You are listened to."


Our culture is one of collaboration, diversity, and helpfulness. That goes for our people internally and how we deal with our clients.


At IDM, you can play a part in educating and assisting real South Africans get out of tough financial situations. You can make a difference with the work you do. Our people believe this too - one said, "I wake up each day knowing I am going to make a difference."

Other key benefits:

We make sure our employees are mentally and physically healthy with Healthy Together campaigns throughout the year. We offer various rewards and awards as tokens of recognition for hard work. We have a vibrant office with easy access to stores and outlets if you need. We offer company benefits - like a half day off on your birthday, medical aid, retirement annuity, and further study opportunities should you meet relevant requirements. * We have a culture of growth, meaning you have access to move both upwards or across in the business if you want to.