Description

Introduction

Modern data engineering involves building both real-time and batch data pipelines to meet diverse analytics and operational needs. This course equips data engineers with the skills to implement, manage, and optimize real-time and batch ETL/ELT workflows using Matillion. It blends foundational practices with advanced integration techniques across various cloud platforms.

Prerequisites

• Experience with Matillion ETL basics

• Understanding of data pipeline architecture

• Familiarity with cloud data warehouses (Snowflake, Redshift, BigQuery)

• Working knowledge of SQL and API-based data sources

Table of Contents

1. Introduction to Real-Time vs. Batch Processing
    1.1 Definitions and Use Cases
    1.2 Choosing the Right Pipeline Strategy
    1.3 Latency, Volume, and Freshness Considerations

2. Batch Pipeline Design in Matillion
    2.1 Data Ingestion Techniques
    2.2 Scheduling and Orchestration
    2.3 Handling Large-Scale Batch Loads

3. Real-Time Integration Concepts
    3.1 Near-Real-Time Data Sources
    3.2 Event-Driven Pipelines
    3.3 Using APIs and Webhooks for Streaming Input

4. Connecting to Real-Time Data Sources
    4.1 Integrating with Kafka, Kinesis, and Pub/Sub
    4.2 Streaming via API Query Components
    4.3 Real-Time Ingestion Design Patterns

5. Pipeline Scheduling and Triggering
    5.1 Batch Scheduler Configuration
    5.2 Event-Triggered Orchestration
    5.3 API-based Invocation for Real-Time Workflows

6. Handling Change Data Capture (CDC)
    6.1 CDC Overview and Benefits
    6.2 Implementing CDC with Matillion
    6.3 Syncing Incremental Updates

7. Data Transformation Best Practices
    7.1 Parallelization in Batch vs. Real-Time
    7.2 Performance Considerations for Streaming
    7.3 Ensuring Consistency and Accuracy

8. Monitoring and Alerting Pipelines
    8.1 Logging Real-Time Jobs
    8.2 Setting Up Notifications and Alerts
    8.3 Monitoring Throughput and Failures

9. Case Studies and Architectures
    9.1 Daily Reporting Pipeline (Batch)
    9.2 IoT Stream Processing (Real-Time)
    9.3 Multi-source Hybrid Pipeline Architecture

10. Tips for Scaling and Maintenance
    10.1 Resource Management
    10.2 Job Modularity and Reusability
    10.3 Long-Term Optimization Strategies

Whether processing bulk data nightly or handling continuous real-time streams, Matillion offers a robust platform for building modern data pipelines. By mastering both batch and real-time capabilities, data engineers can meet evolving business demands with agility, efficiency, and reliability.