Course Syllabus

Module 1: Big Data Concepts & Architecture (4 h)

• Topics: 3Vs (volume, velocity, variety), Hadoop ecosystem (HDFS, YARN, MapReduce, Hive, Pig), Lambda vs Kappa
• Description: Understand foundational big data systems, clusters, and data processing pipelines.
• Features: Guided labs exploring HDFS read/write, cluster architecture, and ecosystem overview.

Module 2: Testing Strategy & Test Planning (4 h)

• Topics: Big data vs traditional testing, test life-cycle, quality dimensions (completeness, consistency)
• Description: Define scope, objectives, FDQ metrics, and formality for big data tests.
• Features: Workshops to draft test plans and traceability matrices.

Module 3: Data Validation (6 h)

• Topics: Source-to-target validation, transformation logic, schema checks, referential integrity
• Description: Use SQL/MapReduce/Hive to validate data across ETL stages.
• Features: Hands-on validation labs with MINUS queries and Hive scripts.

Module 4: MapReduce and Processing Testing (5 h)

• Topics: MapReduce job anatomy, MRUnit testing, custom code validation
• Description: Validate mappers/reducers, job output correctness.
• Features: Unit testing MapReduce jobs and validating intermediate outputs.

Module 5: Big Data Tools Testing (4 h)

• Topics: Hive, Pig, Sqoop, Flume, Spark testing
• Description: Apply tests in higher-level processing tools.
• Features: Exercises covering SQL, Pig Latin scripts, ETL ingestion testing.

Module 6: Performance & Scalability Testing (5 h)

• Topics: Throughput/load, latency, cluster scaling, failover, bottleneck analysis
• Description: Assess system performance under volume and stress.
• Features: Run volume tests, tune Spark jobs, simulate failover.

Module 7: Data Quality & Anomaly Detection (4 h)

• Topics: Accuracy checks, anomaly detection, outlier handling, data profiling
• Description: Ensure data quality via rule-based and statistical techniques.
• Features: Profiling labs and anomaly detection using Spark/Python.

Module 8: Security & Compliance Testing (3 h)

• Topics: Data encryption, metadata audit, privacy compliance (PII, GDPR)
• Description: Validate security, lineage, and governance.
• Features: Hands-on audits of encryption and masked data in pipelines.

Module 9: Integration & Regression Testing (3 h)

• Topics: Pipeline integration, incremental loads, regression validation
• Description: Test continuity across pipelines and version upgrades.
• Features: Regression test suite creation and execution.

Module 10: Automation & CI/CD Integration (3 h)

• Topics: Automating validation with scripts, CI pipeline integration (Jenkins), TestOps
• Description: Operationalize data tests into builds.
• Features: Jenkins jobs using PySpark tests and DVC integration for data version control.

Module 11: Advanced Analytics & ML Pipeline Testing (3 h)

• Topics: Model validation, data drift, cross-validation, ML pipeline QA
• Description: Test predictive pipelines and dataset tuning strategies.
• Features: Data drift detection and model validation labs with ML techniques.

Module 12: Capstone Workshop (4 h)

• Description: Teams execute a full big data pipeline: ingestion, MapReduce/Spark testing, validation, performance assessment, CI integration.
• Features: End-to-end scenario, peer review, expert feedback.