CxO.Observability 0.4.0

CxO.Observability

Shared observability primitives for CxO services. One NuGet package that wires up:

• OpenTelemetry metrics — ASP.NET Core, HttpClient, Runtime, Process instrumentation + Prometheus exporter on /metrics
• OpenTelemetry tracing — ASP.NET Core, HttpClient instrumentation + OTLP exporter, with well-known ActivitySources (CxO.Orders, CxO.Workflows, CxO.Dependencies)
• Standard health endpoints — /health, /ready, /live with tag-based filtering
• Dependency telemetry — HTTP (via DelegatingHandler) and WCF/SOAP (via three interception strategies) with a closed-vocabulary metric schema: cxo_dependency_request_total, cxo_dependency_request_duration_seconds, cxo_dependency_timeout_total, cxo_dependency_health_status
• Structured request logging — Serilog request middleware + a LogContext enricher that stamps ClientIp / UserId / OrderId / WorkflowId / CorrelationId on every request log and on the current Activity

Targets net8.0 and net10.0. Distributed via GitHub Packages (ghcr.io/codaxy) and nuget.org.

Install

dotnet add package CxO.Observability

Or as a PackageReference:

<PackageReference Include="CxO.Observability" Version="0.2.*" />

Quick start — three-line integration

In your service's Program.cs:

using CxO.Observability;

var builder = WebApplication.CreateBuilder(args);

// 1. Register the observability stack (metrics + tracing + dependency telemetry + health checks)
builder.AddCxoObservability("cxo-your-service-name");

// ... your other service registrations ...

var app = builder.Build();

// 2. Activate request logging + LogContext middleware (AFTER auth)
app.UseAuthentication();
app.UseAuthorization();
app.UseCxoObservability();

// 3. Expose Prometheus + health endpoints
app.MapCxoObservabilityEndpoints();

app.MapControllers();
app.Run();

That's it. After this your service emits the platform-standard metric/log/trace fingerprint and can be picked up by the central Prometheus + Grafana + Tempo stack.

What you get, concretely

On GET /metrics

Prometheus-format metrics including:

On GET /health, /ready, /live

Aggregated health check responses. Tag a registration ready for readiness-probe inclusion, live for liveness, or leave untagged — it'll appear in /health but not /ready or /live:

builder.Services.AddHealthChecks()
    .AddNpgSql(pgConnectionString, tags: ["ready"])     // readiness: DB must be reachable
    .AddKafka(kafkaOpts, tags: ["ready"])
    .AddCheck("self", () => HealthCheckResult.Healthy(), tags: ["live"]);

OTLP traces

Sent to the collector endpoint resolved from OTEL_EXPORTER_OTLP_ENDPOINT env var (default http://cxo-otel-collector:4317). Lab environment already has the collector; override in other environments.

Structured request logs

Every request produces a Serilog log line enriched with ClientIp, UserId, OrderId, WorkflowId, CorrelationId, RequestPath, StatusCode. The same properties flow through LogContext to any log lines emitted within the request, and are also stamped as tags on Activity.Current so they appear on your traces too.

Probe paths (/metrics, /health, /ready, /live) log at Verbose to avoid drowning your logs in probe noise.

Custom metrics + tracing

Add service-specific meters and activity sources via the optional hooks:

builder.AddCxoObservability(
    serviceName: "cxo-service-order-management",
    configureMeters: meters => meters.AddMeter("CxO.Orders"),
    configureTracing: tracing => tracing.AddSource("CxO.Orders"));

For the CxO order/workflow/dependency convention, the built-in ActivitySources are already registered:

using var activity = CxO.Observability.Tracing.CxoActivitySources.Orders
    .StartActivity("ProvisionOrder");
activity?.SetTag(CxoActivityTags.OrderId, orderId);

Dependency telemetry — HTTP

Attach the shared DependencyTelemetryHandler to any IHttpClientFactory client and give it a static operation-name resolver (to avoid per-URL cardinality explosion in your metrics):

builder.Services
    .AddHttpClient<KeycloakClient>(c => c.BaseAddress = new Uri(cfg["Keycloak:BaseUrl"]))
    .AddDependencyTelemetry(
        dependencyName: "keycloak",
        resolveOperation: request => request.RequestUri?.LocalPath switch
        {
            "/realms/cxo/protocol/openid-connect/token" => "GetToken",
            "/admin/realms/cxo/users"                   => "ListUsers",
            _                                           => "Other"
        });

Every outbound call against KeycloakClient now emits cxo_dependency_request_total{dependency="keycloak", operation="GetToken|ListUsers|Other", status="ok|4xx|5xx|fault|timeout|error"} plus the latency histogram.

Dependency telemetry — WCF / SOAP

Three integration strategies depending on how your SOAP client is constructed. See docs/integration-guide.md in the repo for full examples; the short version:

// Option — DispatchProxy (recommended for ChannelFactory-based clients)
var channelFactory = new ChannelFactory<ILegacyService>(binding, endpoint);
var proxy = WcfTelemetryDispatchProxy.Create(
    channelFactory.CreateChannel(),
    metrics: dependencyMetrics,
    dependencyName: "legacy-billing-soap",
    resolveOperation: message => message.Headers.Action?.Split('/').Last() ?? "Unknown");

Dependency health checks

Register a health check and the package's IHealthCheckPublisher automatically mirrors its pass/fail state onto the cxo_dependency_health_status gauge so dashboards and alerts see the same truth as the /health endpoint:

builder.Services.AddHealthChecks()
    .AddDependencyCheck(
        name: "keycloak",
        probe: async (sp, ct) =>
        {
            var http = sp.GetRequiredService<IHttpClientFactory>().CreateClient();
            var resp = await http.GetAsync(cfg["Keycloak:HealthUrl"], ct);
            resp.EnsureSuccessStatusCode();
        },
        tags: ["ready"]);

The gauge emits 0 = Unhealthy, 1 = Healthy, 0.5 = Degraded, with a {dependency="keycloak"} label.

Metric names, labels, and the dependency status contract

The package intentionally uses a closed set of values for the status label:

Stick to these values. Adding a new one requires a coordinated update of Grafana dashboards and alert rules (DependencyHighErrorRate regex matches 5xx|fault|error|timeout).

Full reference in docs/dependency-naming-conventions.md.

Configuration surface

appsettings.json requirements

The package itself reads nothing from appsettings.json — service name is a code argument, OTLP endpoint is an env var, and request-log dimensions come from HTTP headers / JWT claims at runtime. You can use CxO.Observability in a service with no appsettings.json at all.

Hard host requirement: Serilog must be registered as the logging provider, because UseCxoObservability() calls UseSerilogRequestLogging(...) internally. Minimum wiring:

using Serilog;

builder.Host.UseSerilog((ctx, _, cfg) => cfg
    .ReadFrom.Configuration(ctx.Configuration)   // reads "Serilog" section from appsettings.json
    .Enrich.FromLogContext()                      // required — lets CxoLogContextMiddleware push properties
    .WriteTo.Console());

Recommended appsettings.json Serilog block (CxO platform convention — Console + Seq):

{
  "Serilog": {
    "Using": ["Serilog.Sinks.Console", "Serilog.Sinks.Seq"],
    "MinimumLevel": {
      "Default": "Information",
      "Override": {
        "Microsoft.AspNetCore": "Warning",
        "Microsoft.EntityFrameworkCore": "Warning",
        "System.Net.Http.HttpClient": "Warning"
      }
    },
    "WriteTo": [
      { "Name": "Console" },
      { "Name": "Seq", "Args": { "serverUrl": "http://cxo-seq:5341" } }
    ],
    "Enrich": ["FromLogContext", "WithMachineName", "WithThreadId"],
    "Properties": { "Application": "cxo-your-service-name" }
  }
}

Sink NuGets referenced above: Serilog.Sinks.Console, Serilog.Sinks.Seq. Swap in whatever sinks your environment uses — none are hard-required by CxO.Observability. See docs/integration-guide.md for the full discussion of env-var vs. config-file trade-offs.

Migrating from hand-rolled OTel wiring

If your Program.cs currently looks like:

builder.Services.AddOpenTelemetry()
    .WithMetrics(m => m
        .AddAspNetCoreInstrumentation()
        .AddRuntimeInstrumentation()
        .AddPrometheusExporter());

app.MapPrometheusScrapingEndpoint();
app.MapHealthChecks("/health");

Replace with:

builder.AddCxoObservability("cxo-your-service-name");
// ...
app.UseCxoObservability();
app.MapCxoObservabilityEndpoints();

You gain: AddProcessInstrumentation (CPU!), HttpClient instrumentation, OTLP tracing, dependency telemetry registration, Serilog LogContext middleware, and /ready / /live tag-filtered endpoints. Nothing you had breaks — any meter you previously registered via configureMeters hook still works.

Drop direct OpenTelemetry.Instrumentation.* / OpenTelemetry.Exporter.* PackageReference lines from your .csproj — they come transitively now.

Versioning

Semantic versioning. Breaking changes (metric names, ActivitySource names, public API shape) bump major. Additive features bump minor. Patch for bug fixes. While on 0.x, the minor version may contain breaking changes until 1.0 — watch the release notes.

Links

• Repo: github.com/codaxy/cxo-shared
• Design doc: docs/implementation-design.md
• Integration guide: docs/integration-guide.md
• Dependency naming conventions: docs/dependency-naming-conventions.md

License

Proprietary — © Codaxy. Consumption restricted to CxO platform services and Codaxy-authorized integrators.