Senior Data Engineer

at Key
Location Cape Town, South Africa
Date Posted Sep 17, 2020
Category IT Jobs
Job Type Full-time


Excellent opportunity to demonstrate your deep technical skills as a member of a top team of IT professionals engaged by one of South Africa’s most successful retail businesses based in Cape Town.

This role functions as a core member of an agile team and you’ll be involved in building and supporting data pipelines and datamarts built off those pipelines, both scaleable, repeatable and secure.

Helping to facilitate gathering data from a variety of different sources, in the correct format, assuring that it conforms to data quality standards and assuring that downstream users can get to that data timeously comes naturally to you.

In your role as a Data Engineering professional, you’ll be responsible for the infrastructure that provides insights from raw data, handling and integrating diverse sources of data seamlessly as well as enabling solutions, by handling large volumes of data in batch and real-time by leveraging emerging technologies from both the big data and cloud spaces.

Additional responsibilities include developing proof of concepts and implementing complex big data solutions with a focus on collecting, parsing, managing, analysing and visualising large datasets.

Your work involves:

  • Architecting Data analytics framework.
  • Translating complex functional and technical requirements into detailed architecture, design, and high performing software.
  • Leading data and batch/real-time analytical solutions leveraging transformational technologies.
  • Working on multiple projects as a technical lead driving user story analysis and elaboration, design and development of software applications, testing, and building automation tools.

Main focus areas:

  1. Development and Operations
  2. Database Development and Operations
  3. Policies, Standards and Procedures
  4. Communications
  5. Business Continuity & Disaster Recovery
  6. Research and Evaluation
  7. Coaching/ Mentoring


  • Minimum 4 years’ Bachelor’s degree in computer science, computer engineering, or equivalent work experience
  • AWS Certification at least to associate level


  • Minimum 5 years Data Engineering or Software Engineering
  • 3-5 years demonstrated experience leading teams of engineers
  • 2+ years Big Data experience
  • 5+ years’ experience with Extract Transform and Load (ETL) processes
  • 2+ years Cloud AWS experience
  • At least 2 years demonstrated experience with agile or other rapid application development methods
  • Agile exposure, Kanban or Scrum
  • 5 years demonstrated experience with object oriented design, coding and testing patterns as well as experience in engineering (commercial or open source) software platforms and large scale data infrastructures.


Must be able to:

  • Create data feeds from ‘on premise’ to AWS Cloud (2 years)
  • Support data feeds in production on break fix basis (2 years)
  • Create data marts using Talend or similar ETL development tool (4 years)
  • Manipulate data using python and pyspark (2 years)
  • Process data using the Hadoop paradigm particularly using EMR, AWS’s distribution of Hadoop (2 years)
  • Develop for Big Data and Business Intelligence including automated testing and deployment (2 years)

Must have:

  • Extensive knowledge in different programming or scripting languages
  • Expert knowledge of data modelling and understanding of different data structures and their benefits and limitations under particular use cases.

Further technical skills required:

  • Capability to architect highly scalable distributed systems, using different open source tools.
  • Big Data batch and streaming tools
  • Talend (1 year)
  • AWS: EMR, EC2, S3 (1 year)
  • Python (1 year)
  • PySpark or Spark (1 year) - Desirable
  • Business Intelligence Data modelling (3 years)
  • SQL (3 years)
Only registered members can apply for jobs.