Job Description

Data Engineer

We are seeking a

Data Integrations / Big Data Engineer

with experience in

Apache NiFi

,

Kafka

, and

enterprise systems

, to design and implement secure, scalable, and real-time data pipelines for a high-profile client implementation project. The ideal candidate will have expertise in managing structured and semi-structured data, integrating legacy and modern systems, and automating data flows across cloud and on-prem environments.

Key Responsibilities:


Design, develop, and maintain data pipelines for ingesting and transforming data from various sources (including APIs, flat files, DBs). Integrate messaging systems like Kafka and RabbitMQ into the pipeline for event streaming and system decoupling. Work closely with Business Analysts and BPEs to translate client requirements into technical data flows. Implement and manage connectors for REST APIs, and file-based batch processes. Ensure data lineage, auditability, and performance tuning of NiFi flows. Deploy and monitor data flows in distributed environments using Zookeeper, Kafka, and ElasticSearch for logging and observability. Strong experience in designing and implementing data integration solutions for real-time streaming data.




Required Skills:


3-5+ years of experience in data integration/engineering roles. Experience with Apache NiFi (flow design, templates, version control, custom processors). Proficient in Kafka, RabbitMQ, and event-driven architectures. Experience integrating with legacy systems ((BAPIs, IDocs, OData). Scripting and transformation in Python, Groovy, or Java. Familiarity with distributed systems (Zookeeper, ElasticSearch, etc.). Experience with CI/CD for NiFi or container-based deployments (Docker, Kubernetes).


Nice to Have:


Knowledge of security practices (TLS, role-based access in NiFi). Hands-on with monitoring tools (Prometheus, Grafana). Familiarity with cloud platforms (AWS/GCP/Azure) for data storage or processing.




Other Requirements:


Eligibility for Top-Level Security Clearance: + Candidates must be eligible to obtain and maintain security clearance at the highest level, in accordance with applicable national security regulations.
On-Site Work Requirement: + This role requires full-time, on-site presence at the client's premises located in Pretoria. Remote or hybrid work arrangements are not applicable.