Cost-Monitoring Middleware for AI and RAG in .NET Core

Implementing AI features using Large Language Models (LLMs) like GPT-4o, Claude 3.5, or Llama 3 introduces a new variable to the operational cost equation: token usage. Unlike traditional APIs where costs are often predictable, LLM costs are probabilistic and volume-dependent.

The Problem: The Hidden Cost of Intelligence

In a RAG (Retrieval-Augmented Generation) system, the cost challenge is amplified. A single user query might trigger a retrieval step that injects thousands of tokens of context into the prompt. Without real-time visibility, several issues can arise:

• Token Burn: A bug in a recursive agent loop can consume thousands of dollars in minutes.
• Lack of Attribution: Businesses cannot easily tell which feature, user, or department is driving AI costs.
• Silent Inefficiency: Unoptimized prompts or over-eager retrieval strategies might be wasting context window space without anyone noticing.

Developers need a way to monitor, budget, and report these costs without littering their business logic with pricing calculations.

The Solution: A Unified Monitoring Middleware

With the release of Microsoft.Extensions.AI (available in .NET 9 and 10), we have a standard set of abstractions for AI services. This allows us to implement a clean middleware pattern using the DelegatingChatClient that intercepts every AI request and response.

Step 1: Defining the Cost Configuration

First, we define a structure to hold the pricing for different models, typically stored in appsettings.json.

public record ModelPricing(decimal PricePerMillionInputTokens, decimal PricePerMillionOutputTokens);

public class AiCostOptions
{
    public Dictionary<string, ModelPricing> ModelRates { get; set; } = new()
    {
        { "gpt-4o", new ModelPricing(5.00m, 15.00m) },
        { "gpt-4o-mini", new ModelPricing(0.15m, 0.60m) }
    };
}

Step 2: Implementing the Cost-Monitoring Middleware

The CostMonitoringChatClient wraps any IChatClient and logs the usage metadata returned by the provider, emitting it to OpenTelemetry via System.Diagnostics.Metrics.

using Microsoft.Extensions.AI;
using System.Diagnostics.Metrics;

public class CostMonitoringChatClient : DelegatingChatClient
{
    private readonly AiCostOptions _options;
    private readonly Histogram<double> _costHistogram;

    public CostMonitoringChatClient(IChatClient innerClient, AiCostOptions options, IMeterFactory meterFactory) 
        : base(innerClient)
    {
        _options = options;
        var meter = meterFactory.Create("Neneos.AI.Monitoring");
        _costHistogram = meter.CreateHistogram<double>("ai.request.cost", "USD", "Estimated cost of AI request");
    }

    public override async Task<ChatCompletion> CompleteAsync(
        IList<ChatMessage> chatMessages, 
        ChatOptions? options = null, 
        CancellationToken cancellationToken = default)
    {
        var response = await base.CompleteAsync(chatMessages, options, cancellationToken);
        
        ProcessUsage(response.ModelId, response.Usage, options);
        
        return response;
    }

    private void ProcessUsage(string? modelId, UsageDetails? usage, ChatOptions? options)
    {
        if (usage == null || string.IsNullOrEmpty(modelId)) return;

        if (_options.ModelRates.TryGetValue(modelId, out var rates))
        {
            var inputCost = (usage.InputTokenCount / 1_000_000m) * rates.PricePerMillionInputTokens;
            var outputCost = (usage.OutputTokenCount / 1_000_000m) * rates.PricePerMillionOutputTokens;
            var totalCost = (double)(inputCost + outputCost);

            // Emit to OpenTelemetry for real-time dashboards
            _costHistogram.Record(totalCost, 
                new KeyValuePair<string, object?>("model", modelId),
                new KeyValuePair<string, object?>("deployment", options?.DeploymentName));
            
            Console.WriteLine($"[AI COST] Model: {modelId} | Cost: ${totalCost:F6}");
        }
    }
}

Step 3: Registration in Program.cs

The beauty of the new .NET AI abstractions is how easily you can chain these clients into a pipeline.

builder.Services.AddChatClient(new AzureOpenAIChatClient(...))
    .Use((inner, sp) => new CostMonitoringChatClient(inner, 
        sp.GetRequiredService<IOptions<AiCostOptions>>().Value,
        sp.GetRequiredService<IMeterFactory>()));

Potential Pitfalls and Dangers

While monitoring is essential, building these systems comes with technical and operational risks:

• Pricing Volatility: AI providers change their pricing frequently. Hardcoding rates is risky; consider using a dynamic configuration provider or an external API for current rates.
• Streaming Complexity: Monitoring costs for streaming responses is significantly harder as you must aggregate partial usage chunks, which can be inconsistent across providers.
• Provider Discrepancies: Not all providers return usage statistics in the same way (e.g., some exclude cached tokens), leading to "estimated" rather than "exact" costs.
• Performance Overhead: While minimal, adding telemetry layers to the hot path of AI calls should be done carefully to avoid introducing latency.

Conclusion: Operational Excellence in the AI Era

Observability is the cornerstone of reliable, production-ready systems. By implementing cost-monitoring middleware, you transform your AI integrations from an unpredictable "black box" expense into a measurable and manageable business asset. This data allows you to make informed decisions about model selection and prompt optimization, ensuring your AI features remain economically viable as they scale.