Data Engineering
AI projects don't fail because the model is wrong. They fail because the data isn't there. This track covers the pipelines, schemas, and governance practices that make AI possible — and explains why most organizations' data readiness is the binding constraint on their AI ambitions.
Courses
Recommended courses
dbt Fundamentals
The official dbt fundamentals course covering data transformation in the warehouse, modeling best practices, testing, documentation, and deployment. dbt has become the standard for analytics engineering pipelines that feed AI systems in regulated industries.
dbt Labs
Data Engineering, Big Data, and Machine Learning on GCP
Google Cloud's specialization on building scalable data pipelines and ML workflows on GCP — covering Dataflow, BigQuery ML, and Vertex AI. Practical for teams standardizing on Google Cloud for enterprise AI data infrastructure.
Coursera / Google Cloud
Preparing for Google Cloud Certification: Cloud Data Engineer
Google Cloud's professional certificate for data engineers — covering BigQuery, Dataflow, Pub/Sub, Cloud Composer, and production pipeline patterns. The most practical cloud data engineering certification for teams building AI-ready data infrastructure.
Coursera / Google Cloud
Books
Essential reading
Designing Data-Intensive Applications
Martin Kleppmann's comprehensive treatment of data systems — reliability, scalability, and maintainability. The gold standard for architects who need to understand distributed systems, replication, and consistency before building AI data pipelines.
Martin Kleppmann
Fundamentals of Data Engineering
The definitive modern guide to data engineering — covering the full data lifecycle, storage systems, ingestion, transformation, and serving layers. The standard reference for architects building AI-ready data infrastructure in regulated enterprises.
Joe Reis, Matt Housley
Machine Learning and Data Science Blueprints for Finance
Applied ML and data science patterns for financial services — covering algorithmic trading, portfolio management, fraud detection, credit underwriting, and risk management. Bridges the gap between data engineering and financial domain requirements.
Hariom Tatsat, Sahil Puri, Brad Lookabaugh
Videos
Watch and learn
Apache Airflow Tutorial for Beginners
A comprehensive introduction to Apache Airflow for data pipeline orchestration — covering DAGs, operators, scheduling, and production deployment patterns. The most-watched Airflow tutorial for data engineers building scheduled pipeline workflows.
Databricks Data + AI Summit Keynotes
Databricks' annual summit keynotes and technical sessions covering Lakehouse architecture, Delta Lake, MLflow, and AI-ready data platforms. The best source for staying current on the tools and patterns used by data engineering teams in regulated industries.
Data Engineering Explained
IBM Technology's clear explanation of modern data engineering — covering data pipelines, lakes, warehouses, and the role of the data engineer in AI-ready organizations. A concise orientation for technical leaders evaluating data infrastructure investments.
QM Signal
Latest from this track
Privacy Failure in Split-LLM Training, The Returned Gradient Nullifies the Decoys
ArXiv cs.LG
→Corporate-Family Resolution Is Not a String-Matching Problem: A Public Benchmark Stratified by Name Visibility
ArXiv cs.LG
→Federated Attack Campaign Detection via Contrastive Encoding of Threat Indicators in Gradient Updates
ArXiv cs.LG
→Privacy, Robustness, and Fairness Trade-offs in Federated Intrusion Detection: Geometric Indistinguishability at the Aggregation Interface
ArXiv cs.LG
→Spruce: Scalable Private Outsourced Retrieval Using Compact Embeddings
ArXiv cs.LG
→Coupled Tensor-Tensor Completion Method with Applications in Drug Repurposing
ArXiv cs.LG
→IDSPACE: A Novel Document Generator for Reliable Evaluation of Digital Identity Verification Systems [Extended Technical Report]
ArXiv cs.LG
→Privacy Leakage in Federated Learning: Gradient-Based Client Identity Inference and Defenses for Inertial Sensing in Vehicular Edge Networks
ArXiv cs.LG
→Privacy-Preserving Topology-Guided Safety for LLM-Based Multi-Agent Systems via Federated Graph Learning
ArXiv cs.LG
→Federated Causal Discovery via Regression-Directed Cumulants
ArXiv cs.LG
→SoK: Where Do Flow Labels Come From? Auditing Label Provenance in Encrypted Traffic Benchmarks
ArXiv cs.LG
→Pushing Forward Multi-Secret-Key Homomorphic Encryption for Private Average Aggregation
ArXiv cs.LG
→Tools & Resources
Tools worth knowing
Prefect
A modern Python workflow orchestration platform for building resilient data pipelines — with scheduling, caching, retries, and event-based automation. A production-ready Airflow alternative favored by teams who prioritize developer experience and observability.
Databricks
The Lakehouse platform built on Apache Spark — unifying data engineering, analytics, and ML in a single governed environment. The platform of choice for regulated-industry enterprises managing large-scale data for AI systems with strict governance requirements.
Apache Airflow
The leading open-source workflow orchestration platform for scheduling and monitoring data pipelines. Used extensively in regulated industries for auditable, repeatable data workflows with full dependency management and failure recovery.