Core Concepts
| Concept | Description | Analogy |
|---|---|---|
| Pipeline | Logical grouping of activities that perform a task | A recipe with multiple steps |
| Activity | A processing step within a pipeline (Copy, Transform, Execute) | A step in the recipe |
| Dataset | Named view of the data (schema, location) | The ingredients |
| Linked Service | Connection string to a data store or compute | The pantry connection |
| Trigger | When and how a pipeline runs (schedule, event) | The timer or doorbell |
| Integration Runtime | The compute infrastructure that runs activities | The kitchen (Azure or on-premises) |
Copy Activity
The Copy Activity is the most commonly used activity — it copies data from a source to a sink (destination). ADF supports 90+ connectors including:
- Azure: Blob Storage, Data Lake, SQL Database, Cosmos DB, Synapse
- On-premises: SQL Server, Oracle, SAP, file systems
- SaaS: Salesforce, ServiceNow, Dynamics 365, REST APIs
- Other clouds: Amazon S3, Google Cloud Storage, Snowflake
No Code Required Copy Activity is configured visually in the ADF Studio — no coding needed for most data copy scenarios. You select source, destination, and column mappings through a GUI. For complex transformations, use Data Flows or compute activities (Spark, Databricks, stored procedures).
Data Flows
Mapping Data Flows provide a visual, no-code way to transform data at scale. You design transformation logic in a GUI — ADF compiles it to Spark and runs it on a managed Spark cluster.
Common transformations:
- Filter, select, derive new columns
- Join multiple datasets
- Aggregate (group by, sum, count)
- Pivot/unpivot
- Lookup — enrich data from a reference dataset
Triggers
| Trigger Type | When It Fires | Use Case |
|---|---|---|
| Schedule | On a fixed schedule (hourly, daily, cron) | Regular batch loads — nightly ETL |
| Tumbling Window | Fixed-size non-overlapping time windows, sequentially | Time-partitioned loads, retry failed windows |
| Event-based (Storage) | When a file arrives in Blob Storage | Process files as soon as they arrive |
| Custom Events | Azure Event Grid event | Trigger on any Azure resource event |
Integration Runtimes
Integration Runtime (IR) is the compute infrastructure ADF uses to run activities:
| IR Type | Where It Runs | Use For |
|---|---|---|
| Azure IR | Managed Azure cloud compute | Cloud-to-cloud data movement and transformation |
| Self-hosted IR | Your own on-premises or VM | On-premises data sources (behind firewall) |
| Azure-SSIS IR | Managed SSIS runtime in Azure | Lifting existing SSIS packages to Azure |
Self-Hosted IR for On-Premises If your source data is on-premises behind a firewall, you install the Self-Hosted Integration Runtime on a server in your network. It establishes an outbound connection to ADF — no inbound firewall rules needed.
Monitoring
ADF has built-in monitoring in the ADF Studio:
- View all pipeline runs — status, duration, errors
- Drill into individual activity runs
- Rerun failed pipelines from the point of failure
- Alert on pipeline failures via Azure Monitor
ADF vs Synapse Pipelines
| Factor | Azure Data Factory | Synapse Pipelines |
|---|---|---|
| Standalone service? | Yes — independent service | Part of Synapse workspace |
| Features | Same engine | Same engine — tightly integrated with Synapse SQL/Spark |
| Choose when | Standalone ETL, no Synapse workspace | Already using Synapse for warehousing/analytics |
AZ-104 Exam Tip Know ADF is for ETL/data integration (not a database). Know it has 90+ connectors. Know Self-Hosted IR for on-premises connectivity. Know the three trigger types (Schedule, Tumbling Window, Event-based). Know Data Flows compile to Spark for transformation.