Data Engineering
AI projects don't fail because the model is wrong. They fail because the data isn't there. This track covers the pipelines, schemas, and governance practices that make AI possible — and explains why most organizations' data readiness is the binding constraint on their AI ambitions.
Courses
Recommended courses
dbt Fundamentals
The official dbt fundamentals course covering data transformation in the warehouse, modeling best practices, testing, documentation, and deployment. dbt has become the standard for analytics engineering pipelines that feed AI systems in regulated industries.
dbt Labs
Data Engineering, Big Data, and Machine Learning on GCP
Google Cloud's specialization on building scalable data pipelines and ML workflows on GCP — covering Dataflow, BigQuery ML, and Vertex AI. Practical for teams standardizing on Google Cloud for enterprise AI data infrastructure.
Coursera / Google Cloud
Preparing for Google Cloud Certification: Cloud Data Engineer
Google Cloud's professional certificate for data engineers — covering BigQuery, Dataflow, Pub/Sub, Cloud Composer, and production pipeline patterns. The most practical cloud data engineering certification for teams building AI-ready data infrastructure.
Coursera / Google Cloud
Books
Essential reading
Designing Data-Intensive Applications
Martin Kleppmann's comprehensive treatment of data systems — reliability, scalability, and maintainability. The gold standard for architects who need to understand distributed systems, replication, and consistency before building AI data pipelines.
Martin Kleppmann
Fundamentals of Data Engineering
The definitive modern guide to data engineering — covering the full data lifecycle, storage systems, ingestion, transformation, and serving layers. The standard reference for architects building AI-ready data infrastructure in regulated enterprises.
Joe Reis, Matt Housley
Machine Learning and Data Science Blueprints for Finance
Applied ML and data science patterns for financial services — covering algorithmic trading, portfolio management, fraud detection, credit underwriting, and risk management. Bridges the gap between data engineering and financial domain requirements.
Hariom Tatsat, Sahil Puri, Brad Lookabaugh
Videos
Watch and learn
Apache Airflow Tutorial for Beginners
A comprehensive introduction to Apache Airflow for data pipeline orchestration — covering DAGs, operators, scheduling, and production deployment patterns. The most-watched Airflow tutorial for data engineers building scheduled pipeline workflows.
Databricks Data + AI Summit Keynotes
Databricks' annual summit keynotes and technical sessions covering Lakehouse architecture, Delta Lake, MLflow, and AI-ready data platforms. The best source for staying current on the tools and patterns used by data engineering teams in regulated industries.
Data Engineering Explained
IBM Technology's clear explanation of modern data engineering — covering data pipelines, lakes, warehouses, and the role of the data engineer in AI-ready organizations. A concise orientation for technical leaders evaluating data infrastructure investments.
QM Signal
Latest from this track
Communication-Efficient Byzantine-Robust Federated Conformal Prediction via Partial Model Sharing
ArXiv cs.LG
→MLQENABLER: Enabling Secure Machine Learning Queries over Encrypted Database in Cloud Computing
ArXiv cs.LG
→Benchmark Evaluation of Feredated Learning on Multi-organ Images
ArXiv cs.LG
→Federated Deep Learning for Privacy-Preserving Cardiovascular Disease Risk Prediction
ArXiv cs.LG
→Workload-Preserving Differentially Private Synthetic Data for Causal Inference via Maximum-Entropy Calibration
ArXiv cs.LG
→Multi-Class vs. Multi-Label BERT for CVE-to-CWE Mapping: How Taxonomy Structure Shapes the Errors
ArXiv cs.LG
→Collaborative Synthetic Data Generation for Knowledge Transfer in Federated Learning
ArXiv cs.LG
→FedCVESA: Taking Away Training Data in Federated Learning via Correlation Value Encoding and Segmented Aggregation
ArXiv cs.LG
→Dual Attention Heads for Personalized Federated Learning in ECG Classification
ArXiv cs.LG
→The foundational elements of AI architecture that IT leaders need to scale
MIT Technology Review — AI
→Reliable Mislabel Detection for Video Capsule Endoscopy Data
ArXiv cs.LG
→REAN: Reconstruction-aware ECG Anonymization Based on Privacy--Utility Orthogonality
ArXiv cs.LG
→Tools & Resources
Tools worth knowing
Prefect
A modern Python workflow orchestration platform for building resilient data pipelines — with scheduling, caching, retries, and event-based automation. A production-ready Airflow alternative favored by teams who prioritize developer experience and observability.
Databricks
The Lakehouse platform built on Apache Spark — unifying data engineering, analytics, and ML in a single governed environment. The platform of choice for regulated-industry enterprises managing large-scale data for AI systems with strict governance requirements.
Apache Airflow
The leading open-source workflow orchestration platform for scheduling and monitoring data pipelines. Used extensively in regulated industries for auditable, repeatable data workflows with full dependency management and failure recovery.