1. Course Syllabus

   1. Module 1: Introduction to Data Engineering & Big Data Ecosystem
      • 1 Big Data Concepts and Hadoop/MapReduce overview
      • 2 Streaming vs. Batch Processing; Use Cases for Real-Time Analytics
      • 3 Survey of Stream Processing Frameworks (Beam, Flink, Storm) and Use Cases
      • 4 Overview of Ecosystem Tools: Apache Kafka, Spark, and Google Dataflow (Beam’s GCP runner)
   2. Module 2: Apache Beam Fundamentals
      • 1 Beam Programming Model: PCollections, PTransforms, and Pipelines
      • 2 SDK Basics (Java/Python): Writing a simple Beam pipeline end-to-end
      • 3 Functional Programming in Beam (Java 8 Lambdas) and Maven/Eclipse setup
      • 4 Hands-on Lab: Build & run a Beam batch pipeline on a local/direct runner
   3. Module 3: Advanced Apache Beam
      • 1 Windows & Triggers: Session, Tumbling, Sliding windows and triggering mechanisms
      • 2 Beam Runners & Google Dataflow: Deploying pipelines on Google Cloud Dataflow
      • 3 Connecting Beam to Data Sources and Sinks (Kafka, Pub/Sub, BigQuery, BigTable)
      • 4 Schema Management with Avro: Using Avro for data schemas in Beam
      • 5 Lab: Streaming Beam pipeline (Kafka → Beam → BigQuery), including unit testing
   4. Module 4: Apache Flink Fundamentals
      • 1 Flink Architecture & APIs: JobManager/TaskManager roles, DataStream vs. DataSet API
      • 2 Stream vs. Batch with Flink: When to use Flink for continuous data processing
      • 3 DataStream Transformations: map, filter, keyBy, and aggregations
      • 4 Hands-on Lab: Develop a basic Flink streaming job (e.g. ingest from Kafka, process events)
   5. Module 5: Advanced Apache Flink
      • 1 Event-Time & Watermarks: Handling out-of-order events and lateness
      • 2 Windows in Flink: Tumbling, Sliding, Session windows for streaming analytics
      • 3 Stateful Processing & Fault Tolerance: State backends, checkpointing, exactly-once guarantees
      • 4 Connectors and Integration: Flink connectors for Kafka, HDFS/S3, Elasticsearch, JDBC, etc.
      • 5 Flink SQL & Table API: Introduction to declarative stream processing
      • 6 Use-Case Lab: Build a Flink pipeline for real-time analytics (e.g. alerting) using Kafka and display results in a dashboard or database.
   6. Module 6: Apache Storm Fundamentals
      • 1 Storm Architecture: Nimbus, Supervisors, and Storm clusters
      • 2 Spouts and Bolts: Core components of a Storm topology; stream groupings (shuffle, fields, all, direct)
      • 3 Reliability & Scalability: Message acknowledgements and fault recovery in Storm topologies
      • 4 Lab: Implement a simple Storm topology (e.g. read tweets from Kafka, perform word count)
   7. Module 7: Advanced Apache Storm
      • 1 Trident API: Micro-batching, stateful stream processing with Storm Trident
      • 2 Integration with Kafka and Databases: Using KafkaSpout, and storing to Cassandra/HBase
      • 3 Case Studies: Real-time analytics with Storm (e.g. Twitter analysis, log processing)
      • 4 Lab: Build a Storm topology using Trident for real-time computation (e.g. continuous computation on streams of data).
   8. Module 8: Ecosystem Integration & End-to-End Pipelines
      • 1 Apache Kafka Deep Dive: Cluster setup, topics, and consumption semantics (to feed Beam/Flink/Storm)
      • 2 Hadoop and Spark in Streaming: Using HDFS/S3 for storage, and optionally Spark Streaming/Structured Streaming as a complementary tool
      • 3 Google Dataflow and Pub/Sub: Deploying Beam on GCP, ingesting from Cloud Pub/Sub, using BigQuery sinks
      • 4 Lab: End-to-end pipeline – e.g. Kafka → Flink (on EMR) → HDFS/S3 → Spark batch job (with integrated visualization).
   9. Module 9: Cloud Deployment and Scalability
      • 1 Google Cloud Platform: Running Beam on Dataflow, setting up Pub/Sub, and monitoring pipelines
      • 2 Amazon Web Services: AWS Managed Flink (Amazon Kinesis Data Analytics), EMR clusters for Storm/Flink, S3 integration
      • 3 Microsoft Azure: HDInsight Storm clusters (Storm on HDInsight), Azure Databricks/Synapse for Spark integration
      • 4 Scalability & Monitoring: Autoscaling, multi-AZ high availability (e.g. AWS Flink’s multi-AZ deployment), and metric dashboards.
      • 5 Lab: Deploy a streaming application on a cloud platform (e.g. Beam pipeline on Dataflow or Flink on AWS) and demonstrate autoscaling.
   10. Module 10: Capstone Project & Case Studies
   • 10.1 Capstone Kickoff: Define a real-world streaming analytics project (e.g. e‑commerce fraud detection, sensor data processing) that uses Beam/Flink/Storm and cloud infrastructure.
   • 10.2 Project Work: Students build, deploy, and test their end-to-end pipelines with mentor support.
   • 10.3 Case Studies: Review of exemplary solutions and architectures from industry.
   • 10.4 Course Wrap-up: Review key concepts, Q&A, and prepare for certification exams (quizzes modeled on certification content).

    

Key Features

  Hands-on Labs & Projects: Each module includes practical labs (e.g. building Beam and Flink pipelines) and guided exercises. Students work with real data streams to reinforce concepts

  Real-World Case Studies: We study industry scenarios (e.g. clickstream analytics, IoT data, fraud detection) to illustrate how Beam/Flink/Storm solve big-data challenges

  Capstone Project: A final capstone task ties together all frameworks: for example, ingest data via Kafka, process it with Beam/Flink/Storm, and store results in BigQuery or HDFS. Participants present their solution at course end.

  Certification Prep: Instructors provide quiz questions and review sessions mirroring certification exams. Labs and the capstone project help prepare for Apache Beam/Flink/Storm certification tests.

  Mentoring & Support: Expert instructors and mentors guide students through problem-solving and career advice. There is personalized feedback on labs and projects, plus optional 1:1 mentoring.