Iceberg metrics by subkanthi · Pull Request #82 · Altinity/ice

subkanthi · 2025-12-04T18:36:12Z

No description provided.

…ile.

…errors.

…lly.

subkanthi · 2025-12-11T17:16:42Z

…_metrics

subkanthi · 2025-12-16T13:54:19Z

There are some metrics that ice is not currently tracking

CommitMetrics
totalFilesSizeInBytes
iceberg_commit_added_delete_files_total

shyiko

Looks good overall. Thank you!

shyiko · 2025-12-18T22:49:45Z

ice/src/main/resources/logback.xml

  <!-- hide "Unable to load metrics class: 'org.apache.iceberg.hadoop.HadoopMetricsContext', falling back to null metrics" -->
  <logger name="org.apache.iceberg.aws.s3.S3FileIO" level="ERROR"/>
+  <!-- hide "Unclosed input stream" warnings from Iceberg's S3InputStream finalizer -->
+  <logger name="org.apache.iceberg.aws.s3.S3InputStream" level="ERROR"/>


isn't this an indicator that we're not doing a good job at closing input streams and need to fix that?

This was coming from iceberg API and it was flooding the logs in a loop, I will get the exception

removed, not able to reproduce it, will add in a separate PR.

shyiko · 2025-12-18T22:55:32Z

ice/src/main/java/com/altinity/ice/cli/internal/metrics/InsertWatchMetrics.java

+
+  // ==========================================================================
+  // Public methods
+  // ==========================================================================


nit: comments like these should be avoided

shyiko · 2025-12-18T23:00:12Z

ice/src/main/java/com/altinity/ice/cli/internal/metrics/InsertWatchMetrics.java

+  private final Counter messageParseErrorsTotal;
+
+  /** Returns the singleton instance of the metrics reporter. */
+  public static InsertWatchMetrics getInstance() {


nit: why not use IoDH for instance lazy loading (and avoid the lock?

changed to IoDH

shyiko · 2025-12-18T23:06:26Z

ice/src/main/java/com/altinity/ice/cli/internal/metrics/InsertWatchMetrics.java

+      "Total number of retry attempts due to failures";
+
+  // SQS errors
+  private static final String SQS_RECEIVE_ERRORS_TOTAL_NAME = "ice_watch_sqs_receive_errors_total";


should we perhaps drop sqs from metric names here and below for consistency with metrics above + in case we decide to add support for other message queue?

Updated to include queue as a filter for grafana dashboard.

shyiko · 2025-12-18T23:26:17Z

examples/scratch/README.md

-
+
+# delete partition
+ice delete nyc.taxis_p_by_day   --partition '[{"name": "tpep_pickup_datetime", "values": ["2024-12-31T23:51:20"]}]' --dry-run=false


\ like the rest

shyiko · 2025-12-18T23:30:46Z

...-catalog/src/main/java/com/altinity/ice/rest/catalog/internal/maintenance/OrphanCleanup.java

+      // Record deletion metrics
+      metrics.recordOrphanFilesDeleted(tableName, deletedCount.get());
+      for (int i = 0; i < failedCount.get(); i++) {
+        metrics.recordOrphanDeleteFailure(tableName);


any reason it's not metrics.recordOrphanDeleteFailure(tableName, failedCount.get())?

shyiko · 2025-12-18T23:35:53Z

ice-rest-catalog/src/main/java/com/altinity/ice/rest/catalog/Main.java

    }

+    // Initialize Iceberg metrics reporter for Prometheus (singleton)
+    PrometheusMetricsReporter metricsReporter = PrometheusMetricsReporter.getInstance();


some Metrics objects are passed around (like here), others are not (like MaintenanceMetrics.getInstance()). Let's settle on one approach

Changed the ones passed to RestCatalogAdapter to use singleton and tested the changes.

shyiko · 2025-12-18T23:39:02Z

ice/src/main/java/com/altinity/ice/cli/internal/cmd/InsertWatch.java

+        builder.endpointOverride(endpoint);
+      }
+    } catch (Exception e) {
+      logger.warn("Failed to parse SQS queue URL for endpoint extraction: {}", e.getMessage());


throw instead of ignore-logging, perhaps?

shyiko · 2025-12-18T23:45:02Z

ice/src/main/java/com/altinity/ice/cli/internal/cmd/InsertWatch.java

+      if (host != null && !host.endsWith(".amazonaws.com")) {
+        URI endpoint = new URI(uri.getScheme(), null, host, uri.getPort(), null, null, null);
+        logger.info("Using custom SQS endpoint: {}", endpoint);
+        builder.endpointOverride(endpoint);


I'd suggest introducing cli option (e.g. --watch-endpoint) that allow to specify a different value from --watch as proposed approach falls apart when using localstack, etc. This way things also can continue to use AWS_ENDPOINT_URL_SQS env var when needed.

shyiko · 2025-12-18T23:47:37Z

ice-rest-catalog/src/main/java/com/altinity/ice/rest/catalog/internal/metrics/HttpMetrics.java

+ */
+package com.altinity.ice.rest.catalog.internal.metrics;
+
+import static com.altinity.ice.rest.catalog.internal.metrics.IcebergMetricNames.*;


https://google.github.io/styleguide/javaguide.html#s3.3.1-wildcard-imports

…leton.

shyiko · 2025-12-22T18:23:05Z

examples/s3watch/test/README.md

+`docker compose up`
+
+Start ice in insert mode.
+`insert flowers.iris -p --no-copy --skip-duplicates s3://bucket1/flowers/iris/external-data/ --watch="http://localhost:9324/000000000000/s3-events"`


this needs to be updated, right?

shyiko · 2025-12-22T19:16:17Z

d8a655c should not be included in this PR as it's unrelated to the rest of the changes

subkanthi · 2025-12-22T19:53:55Z

d8a655c should not be included in this PR as it's unrelated to the rest of the changes

removed.

…tions

subkanthi · 2025-12-22T22:43:41Z

Added

Catalog-level:

iceberg_catalog_tables_total (gauge)
iceberg_catalog_namespaces_total (gauge)
iceberg_catalog_operations_total (counter: create/delete table)
S3 requests (for billing visibility, since S3 Tables charges per request too — pricing):
ice_watch_requests_total or similar

snapshots_count
schema_evolutions_count

xieandrew

Issues I found during testing:

Tables created by insert watch don't increment the iceberg_catalog_tables (count of all tables) metric
The "Scan Metrics" row in Grafana is empty. The corresponding panels are under "Overview" instead.

subkanthi added 7 commits December 3, 2025 13:40

Added iceberg metrics to be exposed in /metrics REST API call.

2504f51

Moved hardcoded strings to IcebergMetricNames, added dashboard json f…

3776ed6

…ile.

Added prometheus HTTP metrics to track request duration and response …

e7859cc

…errors.

Removed extra comments.

6f908f4

Added docker compose for ElasticMQ and configuration to test SQS loca…

32f5adc

…lly.

Added InsertWatchMetrics and updated grafana dashboard.

a6918a3

Added metrics for Maintenance.

18491d0

subkanthi added 5 commits December 11, 2025 12:38

Renamed metrics as it collides with prometheus reserved names.

3ce3466

Update dashboard for metrics.

5d0b79f

Update dashboard for metrics.

99d8d3d

Update dashboard for metrics.

a1edefa

Merge branch 'master' of https://github.com/Altinity/ice into iceberg…

3c7aa44

…_metrics

subkanthi added 6 commits December 16, 2025 12:16

Update dashboard for metrics.

694a633

Update dashboard for metrics.

db27ed9

Update dashboard for metrics.

8bcae99

Added logic to track response size

31532b9

Updated grafana dashboard.

a0c6748

Added documentation for grafana dashboard.

dd2e197

subkanthi marked this pull request as ready for review December 17, 2025 13:54

shyiko requested changes Dec 18, 2025

View reviewed changes

subkanthi added 6 commits December 18, 2025 21:58

Removed wildcard imports, addressed review comments.

f4fd62e

Added cli option to pass SQS override URL.

6930d63

Changed singleton to IoDH

463abb2

Changed singleton to IoDH

34ba594

Removed SQS prefix for insert watch events and added queue label.

50b0645

Removed passing around metricsReporter and modified to accessing sing…

a0b65dc

…leton.

shyiko reviewed Dec 22, 2025

View reviewed changes

Updated README.md for ice insert with ElasticMQ endpoint.

b817287

Updated README.md for running in maintenance mode.

d8a655c

subkanthi added 2 commits December 22, 2025 13:29

Removed README.md changes for running in maintenance mode.

2fb9403

Removed s3InputStream error suppress config in logback.xml

eca20da

Added logic to track catalog, tables, namespaces and alterTable opera…

c668943

…tions

Updated GRAFANA metrics README.

d244ad8

subkanthi marked this pull request as draft January 15, 2026 19:19

xieandrew reviewed Feb 6, 2026

View reviewed changes



		# delete partition
		ice delete nyc.taxis_p_by_day --partition '[{"name": "tpep_pickup_datetime", "values": ["2024-12-31T23:51:20"]}]' --dry-run=false

Conversation

subkanthi commented Dec 4, 2025

Uh oh!

subkanthi commented Dec 11, 2025

Uh oh!

subkanthi commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shyiko left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shyiko commented Dec 22, 2025

Uh oh!

subkanthi commented Dec 22, 2025

Uh oh!

subkanthi commented Dec 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xieandrew left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

subkanthi commented Dec 16, 2025 •

edited

Loading

subkanthi commented Dec 22, 2025 •

edited

Loading