Data Engineer in Edinburgh

Data Engineer

Edinburgh

6-Month contract

Paying up to £55p/h (Inside IR35)

Please note - due to the nature of the work, you will need to hold or be eligible to obtain a high level of UK Security clearance - please only apply if suitable

Key Responsibilities:

• Orchestration ingestion and storage of raw data into structured or unstructured solutions.
• Design, Develop, Deploy and Support data infrastructure, pipelines and architecture.
• Implement reliable, scalable, and tested solutions to automate data ingestion.
• Development of systems to manage batch processing and real-time streaming of data.
• Evaluate business needs and objectives.
• Support the implementation of data governance requirements.
• Facilitate pipelines, which prepare data for prescriptive and predictive modelling.
• Working with domain teams to scale the processing of data.
• Identify opportunities for data acquisition
• Combine raw information from different sources.
• Manage and maintain automated tools for data quality and reliability.
• Explore ways to enhance data quality and reliability.
• Collaborate with data scientists, IT and architects on several projects

Required Skillset:

• Technical expertise in designing, building, and maintaining data pipelines, data warehouses, and leveraging data services.
• Proficient in DataOps methodologies and tools, including experience with CI/CD pipelines, containerisation, and workflow orchestration.
• Familiar with ETL/ELT frameworks, and experienced with Big Data Processing Tools (e.g. Spark, Airflow, Hive, etc.)
• Knowledge of programming languages (e.g. Java, Python, SQL)
• Hands-on experience with SQL/NoSQL database design
• Degree in STEM, or similar field; a master's is a plus
• Data engineering certification (e.g IBM Certified Data Engineer) is a plus