Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[QUERY] Sampling Ratio Issue: Unexpected Number of Samples #46060

Open
gensichen opened this issue Sep 19, 2024 · 3 comments
Open

[QUERY] Sampling Ratio Issue: Unexpected Number of Samples #46060

gensichen opened this issue Sep 19, 2024 · 3 comments
Labels
Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. Monitor - Exporter Monitor OpenTelemetry Exporter needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team question The issue doesn't require a change to the product in order to be resolved. Most issues start as that Service Attention Workflow: This issue is responsible by Azure service team.

Comments

@gensichen
Copy link

Library name and version

Azure.Monitor.OpenTelemetry.Exporter, Version=1.3.0.0,

Query/Question

Description

We are using the Azure.Monitor.OpenTelemetry to push OpenTelemetry (OTel) telemetry to Application Insights. When adjusting the Application Insights sampling ratio, we noticed unexpected behavior in the sampling item count.

Steps to Reproduce

  1. Send 80 requests with a sampling ratio of 100%.
    • Expected Result: 80 samples shown in Application Insights.
    • Actual Result: 80 samples shown (as expected).
  2. Immediately change the sampling ratio to 50% and send another 80 requests.
    • Expected Result: 40 samples shown in Application Insights.
    • Actual Result: 230 samples shown (unexpected).

Observations

  • 80 requests with 100% sampling ratio: 80 samples shown (expected).
  • 80 requests with 50% sampling ratio: 230 samples shown (unexpected, expected 40 samples).

Request for Explanation

Can you please provide an explanation for this discrepancy in the sampling results?

Environment

.NET 8.0

@github-actions github-actions bot added Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. Monitor Monitor, Monitor Ingestion, Monitor Query needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team question The issue doesn't require a change to the product in order to be resolved. Most issues start as that labels Sep 19, 2024
Copy link

Thank you for your feedback. Tagging and routing to the team member best able to assist.

@jsquire jsquire added Service Attention Workflow: This issue is responsible by Azure service team. Monitor - Exporter Monitor OpenTelemetry Exporter and removed Monitor Monitor, Monitor Ingestion, Monitor Query labels Sep 19, 2024
Copy link

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @cijothomas @rajkumar-rangaraj @reyang @TimothyMothra.

@TimothyMothra
Copy link
Contributor

Hi @gensichen,

80 requests with 50% sampling ratio: 230 samples shown

This is not expected.

Can you share your full config? I'd like to see how you are configuring sampling.

We have a query in our public docs which shows how to count telemetry and we expect this should match your config:
https://learn.microsoft.com/en-us/azure/azure-monitor/app/sampling-classic-api#knowing-whether-sampling-is-in-operation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. Monitor - Exporter Monitor OpenTelemetry Exporter needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team question The issue doesn't require a change to the product in order to be resolved. Most issues start as that Service Attention Workflow: This issue is responsible by Azure service team.
Projects
None yet
Development

No branches or pull requests

4 participants