Skip to main content

Advanced Data Engineering


Development Academy of the Philippines
Enrollment is Closed

Course Overview

This course deals with the tools available in the big data ecosystem for extraction, transformation, and loading (ETL), data pipelines, and data storage. This enables participants to use more sophisticated ways to deal with data that has outgrown custom-made Python and SQL scripts. Using videos and hands-on exercises, participants will be exposed to different methods for managing and streaming data, in preparation for business intelligence reporting, analytics, and modeling.

What You Will Learn

Upon completion of this course, the learners are expected to:

  • understand the different popular ETL frameworks and how they are used;
  • learn how Apache Spark, Airflow, and Kafka are used to create real-time big data pipelines; and
  • understand the advantages and disadvantages of using different types of storage and database solutions.

Requirements

You will need a computer or laptop with Microsoft Excel installed. Computer or laptop requirements are:

  • For Windows: Core i3 or better, 4GB RAM or better, MS Excel 2007 or better
  • For MacBook: ideally MS Excel 2013 or newer should be installed (some functions require this version on the Mac). If the version of MS Excel is 2011, download and install StatPlus.

Course Content

Week 1 Recipes for Data Analysis in SQL

7 Videos | 3 Activities

7 Videos

  • Welcome to the course!
  • Introduction to Databases and Database Management Software
  • Data Models and the Entity Relationship Diagram
  • Creating an Entity Relationship Diagram
  • Entity Relationship Diagram to the Relational Model
  • Relational Databases and Anomalies
  • Normalization in Databases

3 Activities

  • Decoding Data Engineering: Word Grid and Brainstorming
  • Pop Quiz: Multiple Choice and Drag and Drop
  • Peer Graded Assignment: Entity Relationship Diagram

Week 2 Review: SQL Fundamentals

7 Videos | 3 Activities

7 Videos

  • CREATE Statement and Data Types
  • Keys in SQL and Relating Relations
  • INSERT INTO Statement
  • SELECT Statement
  • UPDATE Statement
  • DELETE Statement
  • JOINS Statement

3 Activities

  • Exercise: Setting-Up Keys
  • Exercise: Classifying Data Types
  • Peer Graded Assignment: Normalizing Data

Week 3 Dimensional Modeling and Differences between OLTP and OLAP Databases

12 Videos | 3 Activities

4 Videos

  • Introduction to Data Warehousing
  • Data Warehousing Components
  • Introduction to the Dimensional Model
  • Designing the Dimensional Model
  • Dimension Tables
  • Dimension Table 1 Types 1
  • Dimension Table 1 Types 2
  • Dimensional Normal Form
  • Fact Tables and Granularity
  • Fact Table Contents
  • From Normalized to Dimensional Model
  • A Star Schema in an RDBMS

3 Activities

  • Pop Quiz: Fact or Bluff and Labelled Diagram
  • Mind Check: Steps Arrangement and Hungman
  • Quiz

Week 4 From OLTP to OLAP Non-RDBMS Sources

9 Videos |3 Activities

9 Videos

  • What is ETL?
  • Extract and Transform Subsystems
  • Loading Subsystems
  • Managing Subsystems
  • Notes on Transformation
  • Design and Development
  • The ETL Roadmap for Facts
  • Introduction to Non-RDBMS Sources
  • Extracting and Manipulating Data from Non-RDBMS Sources

3 Activities

  • Decoding ETL: Anagram
  • Recall: Steps Arrangement and Multiple Choice
  • Peer-Graded Assignment: SQL Scripts and Coding in Python

Week 5 SQL and ETL Challenges

3 Videos | 1 Activity

3 Videos

  • SQL and ETLChallenges
  • Review on Feature Engineering
  • What is Big Data?

1 Activity

  • Peer Graded Assignment: SQL and ETL Challenges

Week 6 Capstone Project

1 Video | 1 Activity

1 Video

  • SKey Takeaways

1 Activity

  • Capstone Project

  1. Course Number

    SP703
  2. Classes Start

    TBA
  3. Classes End

  4. Price

    Php1000
  5. Prerequisites

    Development Academy of the Philippines SP101

    You must successfully complete Development Academy of the Philippines SP101 before you begin this course.