Add HTTP basic auth and no-auth support for prometheus-sink by ps48 · Pull Request #6595 · opensearch-project/data-prepper

ps48 · 2026-03-02T21:19:45Z

Description

No-auth / plain HTTP support

Make aws config optional by removing @NotNull from awsConfig field in PrometheusSinkConfiguration
Allow http:// URLs — isValidConfig() previously required https:// only, now accepts both http:// and https://
Add isValidAwsConfig() cross-validation that enforces https:// only when aws block is present
Send x-amz-content-sha256 header only when AWS SigV4 signer is configured (was unconditionally sent on every request)
Fix Content-Encoding header value — changed from .toString() (produced SNAPPY) to .name().toLowerCase() (produces snappy), matching the Prometheus Remote Write spec

HTTP Basic authentication

Wire existing AuthenticationOptions and BasicAuthCredentials config classes into PrometheusSinkConfiguration
Add authentication field with @JsonProperty and @Valid annotations
Compute Base64-encoded Authorization header at construction time in PrometheusHttpSender and inject it into outbound requests
Add isValidAuthConfig() cross-validation that rejects mutually exclusive aws + authentication configs (SigV4 and basic auth would conflict on the Authorization header)

README rewrite

Replace placeholder/aspirational documentation with accurate usage examples for no-auth, basic auth, and AMP scenarios
Add structured configuration tables (Required, Optional, Threshold, AWS, Authentication)
Update "Not Yet Implemented" section (remove HTTP Basic auth, keep bearer token, TLS, proxy, etc.)
Fix Java compatibility note (Java 8 → Java 11)
Add build/test/formatting commands

Test coverage

Config deserialization for basic auth fields (username/password round-trip)
Config null check when authentication block is absent
Mutual exclusion validation (aws + auth = invalid, auth-only = valid)
AWS config with http:// URL is invalid
http:// URL without AWS is valid
No-auth sender sends successfully without AWS config
No x-amz-content-sha256 header when AWS config is absent
Basic auth sender sends successfully with 200 response
Captured request has correct Authorization: Basic <base64> header
No Authorization header when authentication is not configured
PrometheusSink initializes with basic auth config
PrometheusSink initializes without AWS config

Issues Resolved

Resolves #6594

Check List

New functionality includes testing.
New functionality has a documentation issue. Please link to it in this PR.
- New functionality has javadoc added
Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: ps48 <pshenoy36@gmail.com>

kylehounslow · 2026-03-12T20:18:57Z

+    }

+    private static String buildAuthHeader(final PrometheusSinkConfiguration config) {
+        if (config.getAuthentication() != null && config.getAuthentication().getHttpBasic() != null) {


Should BasicAuthCredentials.username and BasicAuthCredentials.password have @NotNull annotations (like BearerTokenOptions does for its fields)? As-is, a config with authentication.http_basic: {} would pass validation but seems this line would encode "null:null" into the Base64 header. Could be a follow-up if you want to keep this PR scoped.

Added @NotNull to both username and password fields, preventing authentication.http_basic: {} from silently encoding "null:null" into the Base64 auth header.

kylehounslow · 2026-03-12T20:20:04Z

                .method(SdkHttpMethod.POST)
                .uri(URI.create(url))
-                .putHeader("Content-Encoding", config.getEncoding().toString())
+                .putHeader("Content-Encoding", config.getEncoding().name().toLowerCase())


👍 Good catch

kylehounslow · 2026-03-12T20:21:36Z

+        when(sinkConfig.getAwsConfig()).thenReturn(null);
+        when(sinkConfig.getUrl()).thenReturn("http://localhost:9090/api/v1/write");
+        final byte[] bytes = UUID.randomUUID().toString().getBytes(StandardCharsets.UTF_8);
+        when(httpStatus.code()).thenReturn(200);
+        when(httpData.array()).thenReturn(bytes);
+        when(aggregatedHttpResponse.status()).thenReturn(httpStatus);
+        when(aggregatedHttpResponse.content()).thenReturn(httpData);


Nit: the new tests repeat the same mock setup block (~6 lines each). Could extract a helper to reduce boilerplate. Not blocking.

this should be fine for now. May be something for later improvement when we add TLS support

kylehounslow · 2026-03-12T20:23:30Z

-    @AssertTrue(message = "encoding or content_type or remote_write_version is incorrect.")
+    @AssertTrue(message = "Cannot use both AWS SigV4 and authentication options. Choose one.")
+    boolean isValidAuthConfig() {
+        return !(awsConfig != null && authentication != null);


Should this also reject configs where authentication.bearer_token is set? Since bearer token isn't wired up yet, it would silently fall through to unauthenticated requests. Maybe an additional @AssertTrue validator or a check here?

Added isValidBearerTokenConfig() validator with @AssertTrue that rejects configs where authentication.bearer_token is set, since it's not yet implemented and would silently fall through to unauthenticated.

and add null checks Signed-off-by: ps48 <pshenoy36@gmail.com>

ps48 · 2026-03-13T23:43:43Z

Also tested the AMP support locally making sure AMP ITs still work as expected:

./gradlew :data-prepper-plugins:prometheus-sink:integrationTest \
      -Dtests.aws.region="" \
      -Dtests.aws.role="" \
      -Dtests.prometheus.url="" \
      --tests "*AMPIT*"

> Task :data-prepper-plugins:prometheus-sink:compileIntegrationTestJava
Note: /Users/sgguruda/work/opensource/repos/data-prepper/data-prepper-plugins/prometheus-sink/src/integrationTest/java/org/opensearch/dataprepper/plugins/sink/prometheus/PrometheusSinkAMPIT.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: /Users/sgguruda/work/opensource/repos/data-prepper/data-prepper-plugins/prometheus-sink/src/integrationTest/java/org/opensearch/dataprepper/plugins/sink/prometheus/PrometheusSinkAMPIT.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.8/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

BUILD SUCCESSFUL in 3m 34s
70 actionable tasks: 2 executed, 68 up-to-date

ps48 · 2026-03-16T19:23:45Z

@kkondaka adding the screenshot of the AMP sink test result:

ps48 requested review from KarstenSchnitter, dinujoh, dlvenable, engechas, graytaylor0, kkondaka, oeyh, san81, sb2k16 and srikanthjg as code owners March 2, 2026 21:19

ps48 mentioned this pull request Mar 2, 2026

feat: enable APM RED metrics ingestion into Prometheus via Data Prepper opensearch-project/observability-stack#35

Merged

Add support for basic-auth and no-auth for prometheus sink

5d6ed88

Signed-off-by: ps48 <pshenoy36@gmail.com>

ps48 force-pushed the prometheus-sink-auth branch from 1d9d7a6 to 5d6ed88 Compare March 5, 2026 17:58

kylehounslow reviewed Mar 12, 2026

View reviewed changes

kylehounslow approved these changes Mar 12, 2026

View reviewed changes

resolve comments

faad990

and add null checks Signed-off-by: ps48 <pshenoy36@gmail.com>

kylehounslow mentioned this pull request Mar 16, 2026

feat: Helm umbrella chart for observability-stack kylehounslow/observability-stack#2

Closed

8 tasks

kkondaka approved these changes Mar 19, 2026

View reviewed changes

graytaylor0 approved these changes Mar 19, 2026

View reviewed changes

graytaylor0 merged commit 6d4cfcc into opensearch-project:main Mar 19, 2026
69 of 72 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add HTTP basic auth and no-auth support for prometheus-sink#6595

Add HTTP basic auth and no-auth support for prometheus-sink#6595
graytaylor0 merged 2 commits intoopensearch-project:mainfrom
ps48:prometheus-sink-auth

ps48 commented Mar 2, 2026 •

edited

Loading

Uh oh!

kylehounslow Mar 12, 2026

Uh oh!

ps48 Mar 13, 2026

Uh oh!

kylehounslow Mar 12, 2026

Uh oh!

kylehounslow Mar 12, 2026

Uh oh!

ps48 Mar 13, 2026

Uh oh!

kylehounslow Mar 12, 2026

Uh oh!

ps48 Mar 13, 2026

Uh oh!

ps48 commented Mar 13, 2026

Uh oh!

ps48 commented Mar 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

ps48 commented Mar 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

No-auth / plain HTTP support

HTTP Basic authentication

README rewrite

Test coverage

Issues Resolved

Check List

Uh oh!

kylehounslow Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

ps48 Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

kylehounslow Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

kylehounslow Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

ps48 Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

kylehounslow Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

ps48 Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

ps48 commented Mar 13, 2026

Uh oh!

ps48 commented Mar 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ps48 commented Mar 2, 2026 •

edited

Loading