Web Scraping, and end-to-end data pipelines
Students learn to extract data from HTML and XML using Python libraries and progress to more advanced scraping techniques. The module then covers data ingestion, cleaning, transformation, storage and retrieval, before introducing Apache Airflow for workflow orchestration.
Prerequisites
To take this module you must first have completed Advanced Machine Learning and Programming in Python.
Delivery
Live online lectures and tutorials, conducted at scheduled times and recorded for review.
Study Hours
Approximately 100 hours of study, of which 15 are contact hours and 85 are private study.
Software Used
Python, Apache Airflow
Credits
10 CATS credits
Module Lead
Programme Faculty
Key Outcomes
- —Extract structured data from HTML and XML
- —Use Python web-scraping libraries and frameworks
- —Build ingestion, cleaning and transformation workflows
- —Select suitable data storage and retrieval approaches
- —Orchestrate end-to-end pipelines with Apache Airflow
Lecture Programme
- WK 01
Introduction to web scraping
- ·HTML and XML structures
- ·Extracting web data with Python
- WK 02
Advanced web scraping
- ·Python scraping frameworks
- ·Reliable extraction workflows
- WK 03
Introduction to data engineering
- ·Data ingestion and cleaning
- ·Transformation, storage and retrieval
- WK 04
Apache Airflow and data orchestration
- ·Designing end-to-end pipelines
- ·Scheduling and orchestrating workflows
Assessment
Online coursework submission with support from the module lecturer and tutor.
Build your qualification