Separate AI requests from retries and evaluation work
On this page
This check is due for a source refresh. Confirm the current documentation before you rely on provider-specific details.
Compare application requests with model invocations and evaluation activity before allocating AI spend, because hidden activity can distort attribution. Review retries separately, and do not treat every retry, tool call, or loop iteration as a billed model request.
What you need first
- Tool
- Amazon CloudWatch is AWS's metrics service. Start in its console with the AWS/Bedrock/Agents namespace. For Azure, use telemetry from an existing monitoring gateway, if available. In the documented Azure topology, that gateway can capture client identity or custom identifiers, model, and token data.
- Access
- Ask an authorized AWS colleague for the chosen CloudWatch scope and period, an Azure owner for existing telemetry or evaluation records, and finance for matching billing data. CloudWatch requires appropriate monitoring permissions.
- If you do not use that tool
- Hand the account or subscription, service, region, workload, and start/end times to the workload owner. Request scoped request, model, token, retry, and evaluation results. Ask finance for billing data covering the same scope and period.
Why this is worth a look
Skipping this review can hide model or evaluator activity behind application request counts and misclassify retries as demand. Bedrock agent metrics separate API-operation requests from requests made to a model. AWS SDKs can retry transient or throttled requests. Bedrock judge jobs require an evaluator model and may also use a target model. Foundry can evaluate preexisting outputs without rerunning the target, while AI-assisted quality evaluation requires a deployed GPT model.
Run this check
CHECKLISTUse existing metrics and records for one workload and a fixed time window. This checklist changes no retry settings, deploys no gateway, and submits no evaluation job.
1. Set one matching scope and time window.
Record the AWS account or Azure subscription, service, region, workload, start and end times, and time zone.
Request existing telemetry and evaluation records from authorized workload owners, and matching billing data from finance.
Mark missing records as unknown, not zero.
2. Separate requests from model activity.
In the CloudWatch console, select namespace AWS/Bedrock/Agents for Bedrock agents.
Record InvocationCount, the number of requests to the API operation, and ModelInvocationCount, the number of requests the agent made to the model.
Record InputTokenCount and OutputTokenCount separately. These count metrics use SampleCount; latency metrics use milliseconds.
Record dimensions and time aggregation. Operation, ModelId, and AgentAliasArn are available in different metric combinations.
Review invocation and model throttles and errors separately. Do not add counters together as billed requests.
For Azure, review existing gateway identity or business-unit, tenant, or application identifiers with model and token data. Mark missing attribution as a gap.
3. Review retries and evaluation activity separately.
Ask the AWS workload owner for effective SDK retry_mode and max_attempts settings and existing evidence of retry attempts.
Do not use a retry limit as the observed number of attempts.
For existing Bedrock judge jobs, record the evaluator model, target or generator model if used, prompt dataset, and output bucket.
For Foundry evaluations, record the target, dataset, and evaluators. Distinguish model or agent runs from scoring preexisting outputs.
Do not assume that reused outputs remove evaluator-model activity.
4. Allocate only after resolving scope gaps.
Compare application requests, model activity, and evaluation activity only when scope, period, units, and aggregation match.
Keep AWS and Azure results separate until their units and aggregation are understood.
Ask finance to compare the results with available billing data. Do not convert invocation or retry counts directly into charges.
Record unexplained activity and missing attribution, with an owner for each follow-up.
Keep the review read-only: do not change settings, deploy monitoring, or submit evaluation jobs.How to confirm it
- 01
Choose one workload and period
Record the account or subscription, service, region, workload, and start and end times with time zone. Send the same scope to the workload owner and finance so results can be compared.
- 02
Separate requests from model calls
Compare Bedrock InvocationCount with ModelInvocationCount, then review token counts, throttles, and errors separately. For Azure, use existing gateway identifiers with model and token data, and request missing attribution from the workload owner.
- 03
Account for retries and evaluations
Read the effective AWS retry_mode and max_attempts settings and any existing attempt records. Separate observed attempts from configured limits. Review evaluation targets, datasets, and evaluator models, including runs that score existing outputs.
- 04
Resolve gaps before allocating spend
Ask finance to compare the scoped usage results with available billing data. Assign unexplained model activity or missing attribution to the workload owner. Do not use a model-to-API request ratio as proof of retries or charges.
Before making changes
Assume the records cover the same workload, period, units, dimensions, and aggregation. If they do not match, stop the comparison and request matching results. Bedrock invocation and token metrics describe runtime activity, not a billing mapping. AWS retry behavior depends on SDK and configuration, and the Bedrock metrics describe Amazon Bedrock Agents runtime activity, so verify other service records separately before treating the results as complete workload coverage. Azure gateway attribution is optional and usable here only when already captured.