Course Overview
About Course
Deequ is an open-source library from AWS built on Apache Spark that enables declarative “unit tests” for data—defining, verifying, and monitoring data quality at scale . Users can compute metrics like completeness, uniqueness, size, correlation, and more via Analyzers, and profile datasets to understand distributions and identify anomalies . Automated constraint suggestions streamline test creation, while VerificationSuites allow developers to enforce data contracts through custom or auto-generated Checks. A MetricsRepository persists these metrics over time, enabling monitoring setups—such as exporting to InfluxDB and visualization in Grafana—for trend analysis and SLA management
Deequ integrates natively into Spark pipelines, including support for Scala and PyDeequ, making it production-ready in environments like AWS EMR, Glue, or Databricks . Advanced capabilities include incremental metric computation, anomaly detection, and cross-partition validations. Teams can embed checks within ETL workflows, halt on quality breaches, and uphold governance through reproducible tests and dashboarded reporting.
This 40-hour training equips participants with hands-on labs, multi-language proficiency, observability dashboards, and pipeline automation techniques. By course end, attendees can architect end-to-end data quality frameworks: automated, scalable, and trustworthy—empowering both engineering teams and stakeholders with confidence in data integrity.
-
Course Syllabus
- Introduction & Setup (4 h)
- Overview of Deequ: open‑source Amazon library for data “unit tests” on Spark
- Installation, environment setup (Spark, Java, Maven/SBT, Python wrapper PyDeequ)
- Running your first metrics and constraints on sample dataset (e.g., Amazon reviews)
- Metrics Computation & Profiling (6 h)
- Use of Analyzers and AnalysisRunner for completeness, distinctness, null counts, correlations
- Profiling at scale: interpretable statistical summaries and distributions
- Hands‑on labs on profiling large dataframes
- Constraint Suggestion & Verification (6 h)
- Automated constraint suggestion based on data profiling
- Building VerificationSuites and Checks (Completeness, Uniqueness, Ranges, Cross‑column logic)
- Hands‑on exercises: writing and interpreting constraint results
- Metrics Repository & Monitoring (6 h)
- Introduction to MetricsRepository: storing and querying metrics over time
- Integrating with databases or files (JSON, InfluxDB + Grafana)
- Monitoring data trends and detecting anomalies
- Integrations & Pipeline Automation (6 h)
- Scala & Python (PyDeequ) integration into Spark jobs
- Embedding Deequ in batch pipelines (e.g. EMR, Glue, Databricks) and CI/CD
- Triggering quality checks and halting pipelines on failure
- Advanced Data Quality Use Cases (6 h)
- Anomaly detection on metrics, conditional constraints, cross‑dataset joins
- Incremental metrics computation for partitions
- Complex integrity constraints and relational checks
- Custom Constraints & Extensions (4 h)
- Writing custom analyzers/checks for domain‑specific rules
- Extending MetricsRepository and exporters
- Best Practices & Governance (4 h)
- Data quality SLAs, alerting strategies, results visualization
- Governance frameworks: versioning, failure handling, collaboration
- Training recap, capstone project: full pipeline into production
-
Key Features
Interactive labs with real-world datasets (e-commerce, logs)
Dual-language instruction: Scala + PyDeequ
Visual dashboards using InfluxDB + Grafana
CI/CD demos and pipeline integration with Glue/EMR/Databricks
Automated profiling and anomaly detection
Governance best practices and team enablement
Our Upcoming Batches
At Topskill.ai, we understand that today’s professionals navigate demanding schedules.
To support your continuous learning, we offer fully flexible session timings across all our trainings.
Below is the schedule for our Training. If these time slots don’t align with your availability, simply let us know—we’ll be happy to design a customized timetable that works for you.
Training Timetable
| Batches Online/Offline | Batch Start Date | Session Days | Time Slot (IST) | Fees |
|---|---|---|---|---|
| Week Days (Virtual Online) | Aug 28, 2025 Sept 4th, 2025 Sept 11th, 2025 | Mon-Fri | 7:00 AM (Class 1-1.30 Hrs) | View Fees |
| Week Days (Virtual Online) | Aug 28, 2025 Sept 4th, 2025 Sept 11th, 2025 | Mon-Fri | 11:00 AM (Class 1-1.30 Hrs) | View Fees |
| Week Days (Virtual Online) | Aug 28, 2025 Sept 4th, 2025 Sept 11th, 2025 | Mon-Fri | 5:00 PM (Class 1-1.30 Hrs) | View Fees |
| Week Days (Virtual Online) | Aug 28, 2025 Sept 4th, 2025 Sept 11th, 2025 | Mon-Fri | 7:00 PM (Class 1-1.30 Hrs) | View Fees |
| Weekends (Virtual Online) | Aug 28, 2025 Sept 4th, 2025 Sept 11th, 2025 | Sat-Sun | 7:00 AM (Class 3 Hrs) | View Fees |
| Weekends (Virtual Online) | Aug 28, 2025 Sept 4th, 2025 Sept 11th, 2025 | Sat-Sun | 10:00 AM (Class 3 Hrs) | View Fees |
| Weekends (Virtual Online) | Aug 28, 2025 Sept 4th, 2025 Sept 11th, 2025 | Sat-Sun | 11:00 AM (Class 3 Hrs) | View Fees |
For any adjustments or bespoke scheduling requests, reach out to our admissions team at
support@topskill.ai or call +91-8431222743.
We’re committed to ensuring your training fits seamlessly into your professional life.
Note: Clicking “View Fees” will direct you to detailed fee structures, instalment options, and available discounts.
Don’t see a batch that fits your schedule? Click here to Request a Batch to design a bespoke training timetable.
Corporate Training
“Looking to give your employees the experience of the latest trending technologies? We’re here to make it happen!”
