M14 – Data Engineer

We are looking for experienced data engineers to join our team who will be responsible for supporting the development and maintenance of enterprise data products within our organisation’s data platform.

Requirements

• Design, develop, and maintain data pipelines and ETL processes using AWS services (Glue, Athena, S3, RDS)
• Work with data virtualisation tools like Denodo and develop VQL queries
• Ingest and process data from various internal and external data sources
• Perform data extraction, cleaning, transformation, and loading operations
• Implement automated data collection processes including API integrations when necessary
• Design and implement data models (conceptual, logical, and physical) using tools like ER Studio
• Develop and maintain data warehouses, data lakes, and operational data stores
• Create data marts and analytical views to support business intelligence needs using Denodo, RDS
• Implement master data management practices and data governance standards
• Ensure seamless integration between various data systems and applications
• Implement data security and compliance requirements
• Design scalable solutions for data integration and consolidation
• Develop Python scripts in AWS Glue for data processing and automation
• Write efficient VQL/SQL queries and stored procedures
• Design and develop RESTful APIs using modern frameworks and best practices for data services
• Work with AWS Sagemaker for machine learning model deployment and integration
• Manage and optimise database performance, including indexing, query tuning, and maintenance
• Work in an Agile environment and participate in sprint planning, daily stand-ups, and retrospectives
• Implement and maintain CI/CD pipelines for automated testing and deployment
• Participate in peer code reviews and pair programming sessions
• Create and maintain technical documentation for data models and systems
• Follow industry-standard coding practices, version control, and change management procedures

Tagged as: Change Management, Python, SQL