Job Description

Tech stack:

Proficiency in Python 3.x for data processing and automation.



Experience with AWS Glue for ETL processes.



Strong knowledge of AWS Athena for querying large datasets.



Hands-on experience with AWS Lambda for serverless computing.



Familiarity with AWS EC2 for scalable computing resources.



Expertise in AWS CloudWatch for monitoring and logging.



Proficiency in working with PostgreSQL RDS for database management.



Experience with AWS QuickSight for data visualisation and reporting.



Strong understanding of data ingestion pipelines, particularly for Call Detail Records (CDRs).



Proficiency in Git and GitHub for version control and collaboration, including experience with GitHub CI/CD pipelines.



Any additional responsibilities assigned in the Agile Working Model (AWM) Charter

Duties and Responsibilities:

Design, develop, and maintain data ingestion pipelines for the company's connected fleet, focusing on Call Detail Records (CDRs).



Utilise AWS Glue for ETL processes to transform and load data into the analytics platform.



Implement efficient querying solutions using AWS Athena.



Develop serverless applications and workflows using AWS Lambda.



Monitor and optimise data pipelines using AWS CloudWatch.



Manage and maintain PostgreSQL RDS databases for data storage and retrieval.



Create interactive dashboards and reports using AWS QuickSight.



Extensively leverage Terraform to define, deploy, and manage AWS infrastructure as code, ensuring scalability and reliability.



Use Git and GitHub for version control, collaboration, and maintaining code repositories.



Implement and manage GitHub CI/CD pipelines to automate testing, deployment, and delivery processes.



Ensure data security and compliance with the company's information classification requirements.



Stay updated with the latest data engineering tools, technologies, and industry trends.



Identify opportunities for process improvements and automation to enhance pipeline efficiency.



Collaborate with cross-functional teams to understand data requirements and deliver solutions.



Develop and maintain technical documentation for data engineering processes and solutions.



Bachelor's or Master's degree in Computer Science, Software Engineering, or a related field.



Minimum of 3 years' experience as a Data Engineer.



At least 2 years of experience working with AWS services.



Proven experience in building and maintaining data pipelines for large-scale datasets.



Agile working experience



Between 3 - 5 Years