Course Syllabus

Module 1: Introduction to Big Data & Hadoop

This opening module covers Big Data concepts and the history/motivation for Hadoop. Topics include data variety, volume and velocity challenges, and why distributed systems are needed. We introduce Apache Hadoop – an open-source framework for parallel processing of large datasetsaws.amazon.com. Learners study the master/slave architecture of Hadoop and the roles of NameNode, DataNodes, JobTracker (or ResourceManager), and TaskTrackers. The module discusses Hadoop’s evolution (Hadoop 1.x vs 2.x), including the move from classic MapReduce to YARN, and the benefits of commodity hardware clusteringaws.amazon.com.

Hands-on Lab: Install a simple standalone Hadoop instance, and explore the web UIs (NameNode, ResourceManager) to see cluster status.

Module 2: Hadoop Architecture – HDFS, MapReduce & YARN

This module dives into Hadoop’s core architecture:

• HDFS (Hadoop Distributed File System): Concepts of block storage, replication, and rack awareness. Students learn how HDFS maintains durability by replicating file blocks across DataNodesaws.amazon.com. Practical demos cover HDFS commands to create directories, put/get files, and run hdfs fsck for health checks.
• MapReduce & YARN: Overview of the MapReduce programming model and how Hadoop 2.x replaced the classic JobTracker/TaskTracker with YARN (Yet Another Resource Negotiator). Learners see how MapReduce jobs are split into map and reduce tasks, scheduled by YARN. The module explains how YARN allocates memory/CPU resources dynamically across the clusteraws.amazon.com.
• HDFS High Availability: Introduction to NameNode HA with standby NameNode and quorum-based failover.

Hands-on Lab: Write and submit a sample MapReduce (or Spark) job to process data in HDFS. Simulate DataNode failure and observe HDFS self-healing.

Module 3: Hadoop Ecosystem Components (Hive, Pig, Spark, etc.)

The Hadoop platform includes many ecosystem tools for data processing and analysis. This module surveys:

• Hive: Data warehousing on Hadoop with SQL-like queries.
• Pig: High-level scripting (Pig Latin) for data flows.
• Spark: In-memory distributed computing (compared with MapReduce). Includes running Spark jobs on YARN.
• Oozie: Workflow and scheduler for Hadoop jobs.
• Sqoop & Flume: Data ingestion tools for moving data between Hadoop and relational databases (Sqoop) or streaming sources (Flume).
• Hue, HBase: Intro to the Hue web UI and HBase (NoSQL on Hadoop).

We discuss how these tools run on the cluster (usually on top of YARN/HDFS) and their administrative aspects (e.g., configuring Hive metastore).

Hands-on Lab: Run a Hive query over sample data; ingest a CSV with Sqoop; try a simple Spark job via Spark-shell.

Module 4: Cluster Planning, Installation & Configuration

Proper cluster planning is key. This module covers:

• Hardware & OS Selection: Considerations for CPU, memory, disk, and network topology. JBOD vs RAID, virtualization impact, swap tuning, etc.
• Cluster Sizing: Estimating nodes/cores for given workloads.
• Installing Hadoop: Step-by-step setup on Linux: standalone, pseudo-distributed, and fully distributed modes. Students install Hadoop manually and/or via automation (Ansible or scripts).
• Configuration Files: Deep dive into core-site.xml, hdfs-site.xml, yarn-site.xml, and mapred-site.xml. Learners practice editing settings for namenode directories, datanode data volumes, block size, memory allocation, and replication factor.

Coverage includes deploying Hadoop on Cloudera CDH (using Cloudera Manager) and Hortonworks HDP (using Ambari). For example, one exercise shows how to use Ambari to bootstrap and configure an HDP cluster (Ambari automates installation of Hadoop and HDFS across nodes)nobleprog.monobleprog.mo.

Hands-on Lab: Provision a 3-node Hadoop cluster on Linux VMs or AWS EC2 instances. Configure HDFS permissions, start HDFS/YARN daemons, and verify cluster health.

Module 5: Cluster Administration Commands & Node Management

Administrators need command-line proficiency:

• Cluster Commands: Learn the Hadoop scripts (start-dfs.sh, stop-dfs.sh, hadoop-daemon.sh, yarn-daemon.sh, etc.) to start/stop HDFS and YARN services on individual nodes.
• Data Operations: Using hdfs dfs shell commands to manage files/directories in HDFS (put, get, mkdir, rm, chmod).
• Node Commissioning: How to add new DataNodes or TaskNodes to a running cluster.
• Node Decommissioning: Safely remove nodes (using dfsadmin -decommission) to rebalance data.
• Cluster Changes: Applying configuration changes (editing XML files) and refreshing the cluster (without full restart if possible).

Hands-on Lab: Practice adding a new node: simulate adding a Linux VM to the cluster, configure it as a DataNode, and rebalance HDFS. Then simulate removing a node.

Module 6: Distributions – Cloudera Manager & Hortonworks Ambari

Different Hadoop vendors provide management tools. This module compares them and provides guided practice:

• Cloudera Manager (CM): A comprehensive GUI/CLI for CDH cluster administration. Students learn to use CM to install clusters, manage services (HDFS, YARN, Hive, etc.), and apply configurations across hostscloudera.comcloudera.com. Topics include deploying CM server/agents, using parcels to upgrade software, and automating routine tasks.
• Kerberos & Security with CM: Configure basic TLS and Kerberos realms via Cloudera Manager (setting up KDC, principals, and mapping HDFS and YARN into Kerberos).
• Hortonworks Ambari: The HDP counterpart to CM. We cover Ambari’s web interface and REST API for cluster provisioning and management. Students use Ambari to monitor services, adjust configs, and perform rolling upgrades of components.

This module makes clear how core admin tasks are accomplished in each ecosystem. Cloudera’s platform (CDP/CM) and Hortonworks HDP (Ambari) are compared side-by-side.

Hands-on Lab: Use Cloudera Manager to add a new service (e.g. HiveServer2) to the cluster; use Ambari to change a config and restart a service. Observe how both tools track cluster metrics.

Module 7: Hadoop Security (Kerberos, ACLs, Encryption)

Security is critical in enterprise Hadoop. Topics include:

• Kerberos Authentication: Configure Hadoop to use Kerberos for authentication. Students set up a simple KDC, create Hadoop service principals (for NameNode, DataNode, YARN, etc.), and generate keytabscloudera.com.
• HDFS Permissions & ACLs: Understand POSIX-like permissions in HDFS. Demonstrate setting ACLs (setfacl) on HDFS directories to enforce user/group access controls.
• Encryption: Brief on Hadoop data encryption at-rest (HDFS encryption zones) and in-transit (SSL/TLS for RPC/HTTP).
• Security Tools: Overview of Apache Ranger or Sentry for centralized authorization (HDP/Cloudera respectively). The course discusses their roles but focuses on Kerberos basics.

Hands-on Lab: Create HDFS directories with different owners and permissions. Configure a Kerberized cluster (or simulate it using CM/Ambari) and show that unauthenticated users cannot perform actions.

Module 8: High Availability, Backup & Recovery

To ensure reliability:

• NameNode HA: Configure and test Hadoop’s NameNode High Availability (active/passive using ZooKeeper quorum). Practice failover between NameNodes and recovery steps.
• Secondary NameNode/Snapshot Node: Explain checkpointing of namespace.
• DataNode High Availability: Set up multiple DataNodes and observe HDFS replication. Recover a corrupted block by replicating from healthy copies.
• Rollbacks & Recovery: Use hdfs fsck to detect corrupt files. Use hadoop fs -setrep to correct under-replicated blocks. Demonstrate taking HDFS snapshots and restoring from backup.
• Disaster Recovery: Outline strategies (off-site backups, AWS/Azure geo-redundancy).
• Backup: Use distcp to backup HDFS data between clusters or to cloud storage.

Hands-on Lab: Simulate a NameNode failure: force-stop the active NameNode, trigger failover to the standby, and then restart the original. Use HDFS fsck to find corrupt files and fix them.

Module 9: Monitoring and Troubleshooting

Proactive monitoring keeps clusters healthy. This module covers:

• Logging: Locate and interpret Hadoop logs (NameNode, DataNode, YARN ResourceManager). Learn log rotation and size management.
• Metrics: Use Cloudera Manager/ Ambari dashboards and Ganglia/Nagios to visualize CPU, memory, and disk usage across nodes.
• Alerts: Configure alerts (e.g. via Cloudera Manager) for Node failures, disk full, heartbeats lost, etccloudera.com.
• Performance Tuning: Identify bottlenecks (CPU vs I/O) and adjust parameters (map tasks per node, YARN scheduler settings, HDFS block size). Practice using Hadoop’s balancer utility to rebalance HDFS data when new disks/nodes are addedhortonworks.com.
• Troubleshooting Scenarios: Diagnose common issues (e.g., misconfigured keytabs, wrong permissions, insufficient resources).

Hands-on Lab: Configure a monitoring dashboard (Cloudera Manager or Ambari) and simulate a failure (shutdown a DataNode) to see alerts. Practice analyzing a slow MapReduce job and tuning YARN container sizes.

Module 10: Hadoop in the Cloud – AWS EMR and Azure HDInsight

This module explores Hadoop as a managed cloud service:

• Amazon EMR: Amazon EMR is a managed Hadoop/Spark service that automates cluster provisioning on EC2aws.amazon.com. Students learn to create EMR clusters via the AWS console and CLI, select instance types, and configure autoscaling. We cover using Amazon S3 as the data lake (via EMRFS)aws.amazon.com and storing logs/outputs in S3. EMR’s integration with other AWS services (IAM roles, CloudWatch monitoring, EKS integration for containers) is discusseddatacamp.com. Key features are highlighted (elastic scaling, Spot Instances, and pay-for-use)datacamp.com.
• Azure HDInsight: Microsoft’s Azure HDInsight is a cloud Hadoop/Spark service. Students use the Azure portal to create an HDInsight cluster running Hadoop or Spark. We show how HDInsight runs popular open-source frameworks (Hadoop, Spark, Hive, Kafka, etc.) at enterprise scaleazure.microsoft.com. Integration with Azure Data Lake Storage, Azure Synapse, and Active Directory is explained. Key advantages (fast provisioning, autoscaling, Azure security compliance) are covered. The course points out that HDInsight can be a drop-in replacement if migrating on-prem Hortonworks/Cloudera workloadsazure.microsoft.com.

Hands-on Lab: Launch an EMR cluster on AWS (using a provided AWS account), and run a sample Spark job on it. Then create an HDInsight Hadoop cluster on Azure, and use Hive LLAP to query data stored in Azure Blob Storage.

Module 11: Certification Review and Capstone Project

The final module consolidates learning:

• Certification Prep: Review key topics mapped to Cloudera CCA exam objectives. Practice questions and a mock exam help students identify areas needing review.
• Capstone Lab/Project: Teams or individuals design and implement a mini-project, e.g., deploying a fully functional Hadoop ecosystem. Projects may include: building a 5-node cluster with both Hive and Spark workloads; implementing NameNode HA and simulating failover; migrating data into cloud; or securing a cluster with Kerberos and running queries. Each project is documented and presented for feedback.

Key Features

•  Duration: 40 contact hours (typically delivered as five 8-hour days or equivalent blended schedule).
•  Format: Instructor-led sessions (on-site or virtual) with interactive lectures, demos and guided discussions
•  Hands-On Labs: Extensive lab exercises throughout (building multi-node clusters, using Cloudera Manager/Ambari, running MapReduce/Spark jobs, etc.) to reinforce skills
•  Environments: Virtual machine sandbox and cloud accounts provided for safe experimentation (on-premise VM clusters, AWS, Azure).
•  Certification Support: Curriculum maps to Cloudera CCA (Apache Hadoop) objectives. Practice exams and Q&A review are included. (On completion students may receive a certification voucher or guidance, as availablehortonworks.com.)
•  Audience: System administrators, DevOps engineers, or developers with basic Linux/Unix skills. No prior Hadoop experience is required, but familiarity with Linux and networking is recommendedhortonworks.com.