Find costly Azure Data Factory activity patterns

Why this is worth a look

You can target the wrong cost driver if you compare duration alone. Orchestration is billed by activity run, while activity execution is billed by integration runtime hours. Copy activities on Azure Integration Runtime depend on DIUs and execution duration. Data-flow execution and debugging are billed per vCore-hour, with compute type, vCore count, and duration affecting consumption.

Cost Analysis shows factory costs. Monitor shows consumption for each ADF meter for an individual pipeline run, not the actual charged price. Keep usage and cost separate when choosing a design review.

Complete these read-only steps manually in Cost Analysis and Data Factory Monitor. Use one factory and matching reporting period. Keep money, activity counts, and compute hours separate, and do not start or rerun pipelines.

Azure Data Factory cost pattern review

Preparation
[ ] Record one factory, start and end times, and time zone. Use the same boundaries in both views.
[ ] Arrange at least read access for Azure account cost data. Ask an authorized Data Factory colleague for Monitor results if needed.

Factory costs
[ ] In Cost Analysis, review the selected factory and period. Record cost and currency separately from consumption.

Pipeline consumption
[ ] Open Monitor & Manage on the factory's Azure portal page, or Monitor in the Data Factory left sidebar.
[ ] Set the selected time period and time zone. Monitor defaults to the browser's local time zone.
[ ] Select Refresh to update the pipeline and activity run lists. Do not select rerun.
[ ] For each selected pipeline run, record pipeline name, run ID, status, start, end, and duration (HH:MM:SS).
[ ] Open the consumption icon next to the run. Record each reported ADF meter, consumed amount, and unit.
[ ] Keep activity-run counts, DIU-hours for copy on Azure Integration Runtime, and vCore-hours for data flows separate. Do not add unlike units.
[ ] Click the pipeline name to view activity runs. Record activity names, types, and durations to locate work for review.

Decision
[ ] Mark coverage as selected runs or all relevant runs for the factory and period. Do not treat selected runs as the factory total.
[ ] Compare consumption within each meter across the reviewed runs. Note repeated runs and high consumption, rather than ranking only by duration.
[ ] Choose one pipeline or activity for owner review. Record the consumption pattern and reliability requirement before proposing changes.
[ ] Keep Monitor consumption separate from charged cost. Do not label a difference between sampled runs and factory costs as unassigned usage.

How to confirm it

1. 01

   Choose one factory and period

   Record the factory, start and end times, and time zone in the worksheet. Review that factory and period in Cost Analysis, which is the cost view for Data Factory spending.

2. 02

   Compare consumption by meter

   Open Monitor for the same period and time zone, then open the consumption report for each run you review. Compare amounts within each billing meter, not across different units. Mark whether coverage is selected runs or all relevant runs.

3. 03

   Locate the work behind usage

   Click a pipeline name to inspect its activity runs. Record activity names, types, and durations, then check repeated executions as well as long runs when choosing work for owner review.

4. 04

   Choose a review, not an automatic cut

   Select one workload with recurring or high consumption in the reviewed runs. Ask its owner to assess the workload and reliability requirement before proposing changes. Do not use partial run coverage to allocate the full factory bill.

Primary sources