Data Engineering
A practical program for building reliable batch and streaming data platforms using Python, SQL, Apache Airflow, Kafka, Spark, dbt, Docker, cloud services, data-quality tools, and modern lakehouse technologies.
Career paths
Roles this prepares you for
- Junior Data Engineer
- Analytics Engineer
- ETL Developer
- Data Pipeline Developer
- BI Data Engineer
- Junior Cloud Data Engineer
- Data Operations Engineer
Intended audience
Who it is for
- Learners entering data engineering
- Data analysts moving into engineering
- Software developers transitioning into data platforms
- Professionals responsible for ETL, databases or reporting systems
Curriculum
Module by module
Month 1 — Data Engineering Foundations
- Week 1 — Environment Setup and Engineering Workflows: Linux, command line, Python environments, Git, Docker, data roles and data types
- Week 2 — SQL for Data Engineering: PostgreSQL, modelling, joins, CTEs, windows, transactions, indexes and optimisation
- Week 3 — Python Data Pipelines: API, CSV and database extraction, modular ETL, logging, secrets, idempotency and tests
- Week 4 — Apache Airflow Fundamentals: DAGs, TaskFlow, scheduling, backfills, XComs, retries and scheduled ETL
Project: Dockerized Batch ETL Pipeline Orchestrated with Apache Airflow
Month 2 — Warehousing, Quality and Streaming
- Week 5 — Data Warehouses and Data Lakes: OLTP/OLAP, dimensional modelling, facts, dimensions, SCDs, object storage, Parquet and partitioning
- Week 6 — Transformation, Quality and Governance: dbt, Great Expectations, contracts, validation, lineage, privacy and governance
- Week 7 — Apache Kafka and Event Streaming: brokers, topics, partitions, producers, consumers, offsets, delivery and schema evolution
- Week 8 — Batch and Stream Processing: event time, windows, late data, deduplication, checkpointing and Spark Structured Streaming
Project: Real-Time Event Pipeline with Kafka, PostgreSQL and Monitoring
Month 3 — Big Data, Orchestration and Lakehouse Systems
- Week 9 — Cloud Data Engineering: Azure, AWS, Blob Storage, ADLS, S3, IAM, networking, secrets and cost-aware architecture
- Week 10 — Spark and PySpark: architecture, DataFrames, lazy evaluation, partitioning, joins, Spark SQL and performance
- Week 11 — Advanced Apache Airflow: dynamic mapping, branching, sensors, pools, datasets, deadlines, custom operators and production deployment
- Week 12 — Lakehouse Architecture: Delta Lake, ACID, schema evolution, time travel, medallion layers and Iceberg concepts
Project: Cloud-Based Lakehouse Pipeline with Airflow, Spark and Delta Lake
Month 4 — Production Engineering and Capstone
- Week 13 — Production Batch Pipeline: incremental ingestion, CDC, validation, retries, backfills, audit tables and tests
- Week 14 — Production Streaming Pipeline: real-time ingestion, transformations, resilience, duplicate prevention, errors and monitoring
- Week 15 — Observability and CI/CD: Grafana, Prometheus, logs, metrics, alerts, health checks, GitHub Actions and IaC fundamentals
- Week 16 — Capstone Presentation: architecture review, load and failure tests, documentation, runbooks, deployment and portfolio review
Project: Unified Retail Intelligence Data Platform
Capstone project
Unified Retail Intelligence Data Platform
What you build
- Ingests batch sales and inventory data
- Processes real-time order events
- Stores raw data in an object-storage data lake
- Uses Spark to create bronze, silver and gold layers
- Uses dbt for analytics transformations
- Applies Great Expectations data-quality checks
- Uses Airflow for orchestration
- Uses Kafka for streaming
- Publishes business-ready warehouse tables
- Provides Grafana operational monitoring
- Includes automated tests and CI/CD
What you hand in
- SQL dimensional model
- Python ETL pipeline
- Airflow DAGs
- Kafka producer and consumer
- Spark processing application
- dbt project
- Data-quality reports
- Lakehouse implementation
- Monitoring dashboard
- Architecture diagram
- Deployment guide
- Operations runbook
- Final GitHub repository
Assessment
How your progress is assessed
- Weekly practical assignments
- Technical articles and research
- Mini-projects
- Code and project reviews
- Monthly build assessments
- Peer presentations
- Capstone milestones
- Final technical presentation
- GitHub portfolio quality
- Documentation and communication
FAQ
Frequently asked questions
How long are the full data programmes?
A typical journey is 24 weeks: 4 months of instructor-led learning followed by a 2-month internship. Students who need longer usually finish within 8–9 months.
What do the data programmes cost?
Online: KES 7,500 per month (KES 30,000 total). In person: KES 12,500 per month (KES 50,000 total). Paying upfront saves 10%.
When is the next intake?
The next full Data Science & Analytics and Data Engineering intake begins in January 2027.
What is the difference between Data Science & Analytics and Data Engineering?
Data Science & Analytics focuses on analysing data and building predictive models. Data Engineering focuses on the pipelines, warehouses and streaming systems that make data reliable.
How is Machine Learning here different from the School of Artificial Intelligence?
In this school, machine learning is taught as data-driven modelling and analytical methods. The School of Artificial Intelligence focuses on using and integrating AI tools in everyday work.
Is the certificate accredited?
Certificates are issued by LuxDevHQ. They are not externally accredited.
More from the School of Big Data and Analytics
Data Science & Analytics
Open for ApplicationsExcel, SQL, Python, Power BI, statistics, machine learning and applied AI, ending in a portfolio capstone.
- Duration
- 4 months of instruction + 2-month internship
- Next intake
- January 2027
Python and SQL Short Courses
Open for ApplicationsOne-month, instructor-led short courses in Python and SQL for data analysis.
- Duration
- 1 month
- Next intake
- Contact admissions
Data Analytics and Business Intelligence
Coming SoonA dedicated analytics and BI track. Excel, SQL and Power BI are taught today within Data Science & Analytics.
View details