An exciting opportunity to be part of a technology and analytics driven consulting company whose core capabilities span the disciplines of Data Science, Data Engineering and Software Engineering.
Your responsibilities will include:
· Analyse and organise raw data
· Design data engineering solutions to meet business requirements
· Build scalable data pipelines that clean, transform and aggregate data from different sources using appropriate tools and technologies
· Collaborate with data scientists to prepare data sets for analytical modelling
· Identify ways to enhance data quality and reliability
· Manage technical delivery of projects
· Mentor junior data engineers
Qualifications and experience needed:
· 8+ years of relational SQL database
· Solid understanding of Apache Spark as a distributed data processing framework
· Experience in designing, implementing and performance tuning solutions in Apache Spark
· Experience in complex data warehouse and data lake design
· Experience in building, optimising and manging data pipelines for large and complex datasets
· Experience using AWS cloud data platform technologies:
o Amazon S3
o Amazon Redshift
o AWS Glue
o Amazon Athena
o Amazon Elastic MapReduce (EMR)
o AWS Step Functions
o AWS Lambda
to implement data pipelines and analytical solutions
· Experience in Python would be beneficial
· Experience with streaming technologies such as Spark streaming or Kafka would be beneficial
· A relevant data engineering certification would be beneficial
- Graduate or Postgraduate degree in Computer Science, Information Technology, Statistics, Engineering or related technical field.