Course Syllabus

Module 1: Big Data & Hadoop Fundamentals (2 hours)

• Learning Objectives: Understand what “big data” means and why Hadoop was developed. Grasp Hadoop’s place in modern data architectures.
• Key Topics: Big Data characteristics (volume, velocity, variety); limits of traditional databases; Hadoop’s core components (HDFS for storage, MapReduce and YARN for processing); overview of Hadoop ecosystem (Hive, Pig, Spark, etc.). Example use-case: analyzing petabytes of retail transactions that would overwhelm a single database.

Module 2: Hadoop HDFS Architecture (3 hours)

• Learning Objectives: Describe HDFS design and how it stores data across a cluster. Operate HDFS through examples.
• Key Topics: HDFS master-slave architecture: the NameNode (metadata manager) and DataNodes (block storage).  Concepts of file to block mapping, replication factor, and fault tolerance. Hands-on: put/get files in HDFS, replicate data, and browse the HDFS directory structure.
• Industry Example: Storing large log or image files in HDFS to enable parallel analysis.

Module 3: MapReduce Programming (4 hours)

• Learning Objectives: Explain the MapReduce processing model and write basic MapReduce programs.
• Key Topics: The Map and Reduce phases; data flow (mapper output sorting and shuffling, reducer aggregation); writing MapReduce jobs in Java (or another language). Using Hadoop’s command-line to run a job on the cluster. Hadoop processes data by splitting a job into map and reduce tasks.
• Hands-on: Write a word-count and a simple analytics MapReduce job on a sample dataset. Test on multi-node cluster.

Module 4: Advanced MapReduce & Data Flows (4 hours)

• Learning Objectives: Develop complex MapReduce workflows and optimize performance.
• Key Topics: Advanced features: combiners, custom partitioners, counters, and distributed cache. Chaining multiple MapReduce jobs. Best practices for performance tuning (compression, memory settings). Debugging and logging in Hadoop.
• Hands-on: Implement a multi-stage MapReduce pipeline (e.g. parsing and joining data). Optimize a slow job using a combiner.

Module 5: Hadoop YARN (1 hour)

• Learning Objectives: Understand YARN’s role in Hadoop 2.x and how it manages resources.
• Key Topics: YARN architecture: ResourceManager and NodeManager roles. How jobs are submitted and scheduled across the cluster. Difference between Hadoop 1 (old MapReduce) and Hadoop 2 (YARN) models.
• Example: Monitor cluster resources via YARN’s Web UI and submit a sample Spark job under YARN.

Module 6: Hive Data Warehousing (4 hours)

• Learning Objectives: Query and analyze Hadoop data using Hive’s SQL-like language.
• Key Topics: Hive overview: SQL querying on HDFS data. HiveQL syntax, creating databases and tables (text, Parquet formats), and loading data. Partitions and bucketing for performance. How Hive translates queries into MapReduce (or Tez/Spark) jobs.
• Hands-on: Write HiveQL queries to aggregate retail sales data. Create partitioned tables and measure query performance. (Cloudera exam note: Hive skills are tested in certification tasks.)

Module 7: Pig Latin Scripting (2 hours)

• Learning Objectives: Use Apache Pig to build data transformation scripts.
• Key Topics: Pig overview: Pig Latin language for Hadoop. Comparison with Hive (more procedural). Loading, transforming, and storing data with Pig operators. Piggybank UDFs.
• Hands-on: Write a Pig script to clean and filter large log files on HDFS. Use Pig to prepare data for a Hive table.

Module 8: HBase NoSQL Database (2 hours)

• Learning Objectives: Explain HBase’s data model and use HBase for random access to big tables.
• Key Topics: HBase is “the Hadoop database, a distributed, scalable, big data store”. Differences between HBase and Hive/Pig. HBase table design: tables, column families, and row keys. HBase architecture: HMaster and RegionServers. CRUD operations with the HBase shell or Java API.
• Hands-on: Create an HBase table for patient records (e.g. patient_id as key) and insert a few rows. Scan and get records to practice random reads.

Module 9: Data Ingestion with Sqoop (1 hour)

• Learning Objectives: Transfer data between Hadoop and relational databases.
• Key Topics: Sqoop use-case: importing data from RDBMS (MySQL, Oracle) to HDFS or Hive. Sqoop export to move results back to a database. Sqoop commands and options (target-dir, split-by).
• Hands-on: Use Sqoop to import a sample sales table from MySQL into Hive. Demonstrate incremental import.

Module 10: Data Ingestion with Flume (1 hour)

• Learning Objectives: Stream data into Hadoop using Flume agents.
• Key Topics: Flume overview: architecture (Sources, Channels, Sinks). Setting up a Flume agent to collect log data (e.g. web server logs) and deliver to HDFS. Flume reliability and failover.
• Hands-on: Configure a Flume agent to write streaming event data into HDFS.

Module 11: Spark Core (5 hours)

Spark is a fast, in-memory data processing engine supporting Java, Scala, Python, and R. This module introduces Spark’s architecture and RDD (Resilient Distributed Dataset) model. Participants learn to create RDDs from HDFS data, apply transformations (map, filter, join) and actions (collect, save). Key concepts include lazy evaluation, lineage graphs, and in-memory caching.

• Learning Objectives: Build Spark applications and understand performance advantages (in-memory compute) over MapReduce.
• Key Topics: SparkContext and job execution. Common transformations/actions. Building a simple Spark program. Contrast Spark versus traditional MR.

Module 12: Spark SQL & DataFrames (3 hours)

• Learning Objectives: Use Spark’s high-level APIs for structured data processing.
• Key Topics: Spark SQL introduction: DataFrames and Datasets APIs. Reading data into DataFrames (from JSON/Parquet). SQL queries on DataFrames, integration with Hive metastore. Writing results back to HDFS. Spark’s Catalyst optimizer.
• Hands-on: Load a JSON healthcare dataset into a DataFrame, run Spark SQL queries (e.g. aggregate patient records), and compare with Hive performance.

Module 13: Spark Streaming (3 hours)

• Learning Objectives: Implement real-time streaming analytics in Spark.
• Key Topics: Spark Streaming basics: DStreams/micro-batches. Creating streaming contexts and processing live data (e.g., Twitter feeds or socket streams). Windowed operations and stateful streaming.
• Hands-on: Build a Spark Streaming job to count keywords in a simulated log stream. Use checkpointing for fault tolerance.

Module 14: Workflow Scheduling with Oozie (1 hour)

• Learning Objectives: Orchestrate complex Hadoop jobs in production.
• Key Topics: Apache Oozie overview: workflow and coordinator jobs. Defining a workflow XML to chain MapReduce, Hive, and Pig jobs. Time-triggered (coordinator) workflows for periodic data processing.
• Hands-on: Create an Oozie workflow that first runs a Pig script then a Hive query, triggered daily.

Module 15: Real-World Case Studies & Capstone (2 hours)

• Learning Objectives: Apply learned skills to end-to-end use cases drawn from industry.
• Key Topics: Case Study 1: Retail: Build a Hive-based analytics pipeline on sales data for customer segmentation (demonstrating retail personalization). Case Study 2: Healthcare: Analyze patient data with Spark (predictive analytics for readmission risk). Case Study 3: Finance: Create a MapReduce or Spark job for transaction fraud detection. Each project integrates multiple tools (HDFS, Hive/Pig, Spark, etc.) to solve a business problem.
• Outcome: Students present their solutions, showcasing how Hadoop technologies address real data challenges.

Module 16: Certification Review & Next Steps (2 hours)

• Learning Objectives: Review key topics and exam-style tasks; plan certification preparation.
• Key Topics: Recap of important concepts for Cloudera CCA and Hortonworks HDPCD developer exams. Example review questions on writing Hive queries, Pig scripts, or Spark programs. Discussion of exam format (hands-on tasks). Guidance on continuing practice, resources, and career paths in Big Data.

Key Features

• The course is instructor-led (in-person or virtual) with live lectures, demos and Q&A. It emphasizes hands-on labs and projects using real datasets, following the approach advocated by Cloudera and Hortonworks. Each module includes practical exercises: for example, students might build a Hive query engine on a retail sales dataset or a Spark analytics job on healthcare data. Case studies from industries such as retail (customer segmentation and product recommendation), healthcare (patient data analytics), and finance (fraud detection) reinforce concepts.The curriculum explicitly maps to industry certification objectives, offering exam guidance for Hadoop developer tracks.
• Hands-on Labs: Each topic includes instructor-guided labs (Cloudera notes instructor-led courses include lab exercises)where students work on Hadoop clusters and write code.
• Real-world Projects: Practical projects simulate scenarios (e.g. processing retail transaction logs with MapReduce, building a Spark ML workflow on health data). Industry use-case discussions ensure learning is contextual (hashstudioz reports Hadoop aids retail recommendation and patient analytics).
• Comprehensive Curriculum: Covers Hadoop fundamentals and ecosystem end-to-end. Topics range from storing data in HDFS to writing MapReduce jobs, and extend to Hive SQL, Pig scripts, HBase databases, data ingestion (Sqoop/Flume), Spark (core, SQL, streaming), and workflow scheduling (Oozie).
• Certification Alignment: Content aligns with Cloudera Certified Associate (CCA) and Hortonworks HDP Developer exam domains. Instructors highlight key exam concepts (for example, Hortonworks notes Pig, Hive, Sqoop, Flume as core skills). Practice questions and tips help students prepare for certification.
• Structured Delivery: The 40-hour program is divided into modules with fixed durations (see below). Each module has clear objectives and takeaways. The format balances theory with hands-on coding so beginners build confidence quickly.