[QUESTION]

[vpnicholls]

Which component is your question about?

Lite

Performance Monitor Version

2.4.1, 2.3.0

Is your question about how it works, or the results?

How the tool works

What's your question?

Just wondering if anyone has noticed this behaviour. I've scanned through other questions and issues but haven't seen anything.

After installing 2.4.1, I could dismiss alerts, either select alert or all alerts. At some stage there seems to be a tipping point where I neither of the dismiss buttons works. If I click the dismiss all button, then the Dismiss All Alerts dialog box appears. I click OK, then nothing.

I can't figure out the pattern. The underlying cause for each has since resolved. The alerts are for two different instances (no Azure SQL Databases) and two different Alert Types (Blocking Detected and Long-Running Query). I can't recall if there were any other types previously. When it was working, there were fewer alerts, about 4 or 5. When I first noticed it not working there were about 20-30. Right now, I have 35 alerts over 30 hours that I can't dismiss.

I've checked the logs, but I can't see anything around the relevant times when I attempt to dismiss the alerts.

This is happening for me in v2.4.1, but I first noticed this in 2.3.0.

Additional Context

No response

[erikdarlingdata]

@vpnicholls there may be some tie-in to this and the work done by @HannahVernon on the alert muting in #512 and #642. This code path is somewhat hard for me to test thoroughly with my limited home environment set up.

[HannahVernon]

I'm investigating.

[vpnicholls]

Thanks @erikdarlingdata and @HannahVernon. I'll have a read of #512 and #642 .

[HannahVernon]

@erikdarlingdata - I've added alert dismissal / mute logging into a new fix branch which I'll push to you shortly for review. Before I do that, I wanted to present some options we could implement to make Alerts more feature rich and diagnosable, both for the user and for those supporting the product. Take a gander below, and let me know if you think we should implement everything, or just certain things, or nothing at all ☘️🦈🚀

Issue #718 — Alert Dismiss Stops Working: Root Cause Analysis & Suggested Fixes

Root Cause

The Lite dismiss flow has a read/write mismatch between the view and the live table:

• GetAlertHistoryAsync reads from v_config_alert_log — a view that unions the live config_alert_log table with archived parquet files via UNION ALL BY NAME SELECT * FROM read_parquet(glob)
• DismissAlertsAsync / DismissAllVisibleAlertsAsync issue UPDATE config_alert_log SET dismissed = TRUE — targeting only the live table

When ArchiveService.ArchiveOldDataAsync() runs (default: data > 7 days), it COPYs rows to parquet then DELETEs them from the live table under an exclusive write lock. After archival, those alerts still appear in the grid (via the view's parquet union), but the UPDATE can never reach them because they only exist in read-only parquet files. The 0-rows-affected result is silently swallowed.

The ~20–30 alert threshold corresponds to crossing the hotDataDays boundary, especially when the user selects "Last 7 Days" or "All" as the time range, or when size-pressure archival uses hotDataHours for more aggressive cutoffs.

There's also a race condition: DismissAlertsAsync acquires a read lock (via OpenConnectionAsync), but UPDATE is a write operation running under that read lock. If archival is waiting for its write lock, it can proceed the moment the dismiss's read lock releases — potentially archiving rows between the user's dismiss click and the UI refresh.

The Dashboard edition is not affected because it uses in-memory lists with Hidden flags.

---

Suggested Improvements

1. Add a source column to the archive view

Files: Lite/Database/DuckDbInitializer.cs, Lite/Services/LocalDataService.AlertHistory.cs, Lite/Controls/AlertsHistoryTab.xaml, AlertHistoryRow model

Modify CreateArchiveViewsAsync() so the v_config_alert_log view includes a computed origin column:

CREATE OR REPLACE VIEW v_config_alert_log AS
    SELECT *, 'live' AS source FROM config_alert_log
    UNION ALL BY NAME
    SELECT *, 'archive' AS source FROM read_parquet('{globPath}', union_by_name=true)

Add public string Source { get; set; } = "live"; to AlertHistoryRow and map it in GetAlertHistoryAsync. In the XAML, add a DataTrigger on Source == "archive" to grey out the row or show an archive icon. Disable selection of archive-sourced rows for dismiss, or show a tooltip: "This alert has been archived and cannot be dismissed."

This completely prevents the user from attempting an impossible operation.

---

2. Implement parquet-aware dismissal via a sidecar table

Files: Lite/Database/Schema.cs, Lite/Database/DuckDbInitializer.cs, Lite/Services/LocalDataService.AlertHistory.cs

Create a small sidecar table to track dismissed archived alerts:

CREATE TABLE IF NOT EXISTS dismissed_archive_alerts
(
    alert_time    TIMESTAMP NOT NULL,
    server_id     INTEGER   NOT NULL,
    metric_name   VARCHAR   NOT NULL,
    dismissed_at  TIMESTAMP NOT NULL DEFAULT current_timestamp
)

When DismissAlertsAsync detects a 0-rows-affected UPDATE (parquet-only row), INSERT into dismissed_archive_alerts instead. Modify the view's parquet half to filter these out:

SELECT *, 'archive' AS source
FROM read_parquet('{globPath}', union_by_name=true) p
WHERE NOT EXISTS (
    SELECT 1 FROM dismissed_archive_alerts d
    WHERE d.alert_time = p.alert_time
    AND   d.server_id  = p.server_id
    AND   d.metric_name = p.metric_name
)

This makes archived alert dismissal actually work.

---

3. Filter archived rows out of dismiss candidates

Files: Lite/Services/LocalDataService.AlertHistory.cs

Before the UPDATE loop in DismissAlertsAsync, run a quick existence check against the live table:

// Build a set of alerts that still exist in the live table
var liveKeys = new HashSet<(DateTime, int, string)>();
using var checkCmd = connection.CreateCommand();
checkCmd.CommandText = "SELECT alert_time, server_id, metric_name FROM config_alert_log WHERE dismissed = FALSE";
using var reader = await checkCmd.ExecuteReaderAsync();
while (await reader.ReadAsync())
    liveKeys.Add((reader.GetDateTime(0), (int)reader.GetInt64(1), reader.GetString(2)));

Skip the UPDATE for alerts not in liveKeys and collect them into a separate "archived" list returned to the caller. This avoids N futile UPDATEs and provides the information needed for user feedback.

---

4. Upgrade dismiss operations to use a write lock

Files: Lite/Services/LocalDataService.cs, Lite/Services/LocalDataService.AlertHistory.cs

DismissAlertsAsync and DismissAllVisibleAlertsAsync perform UPDATEs but acquire only a read lock via OpenConnectionAsync(). Add a separate OpenWriteConnectionAsync() method that acquires AcquireWriteLock():

internal async Task<LockedConnection> OpenWriteConnectionAsync()
{
    var writeLock = _duckDb.AcquireWriteLock();
    try
    {
        var connection = _duckDb.CreateConnection();
        await connection.OpenAsync();
        return new LockedConnection(connection, writeLock);
    }
    catch
    {
        writeLock.Dispose();
        throw;
    }
}

Use this in both dismiss methods. This prevents the race where archival can interleave with dismissals and ensures the UPDATE doesn't compete with other writers.

---

5. Wrap multi-alert dismissals in a transaction

Files: Lite/Services/LocalDataService.AlertHistory.cs

DismissAlertsAsync loops through N alerts executing individual UPDATEs with no transaction. If the connection drops mid-loop, some alerts are dismissed and others aren't. Wrap the loop:

using var transaction = await connection.BeginTransactionAsync();
try
{
    foreach (var alert in alerts)
    {
        // ... existing UPDATE logic ...
    }
    await transaction.CommitAsync();
}
catch
{
    await transaction.RollbackAsync();
    throw;
}

This ensures all-or-nothing semantics and reduces write lock hold duration.

---

6. Show user-visible feedback on partial/failed dismissals

Files: Lite/Controls/AlertsHistoryTab.xaml.cs, Lite/Controls/AlertsHistoryTab.xaml

Currently, partial failures are logged to a file the user will never check. When affected < selected.Count, show an info bar or message:

if (affected < selected.Count)
{
    var archived = selected.Count - affected;
    MessageBox.Show(
        $"Dismissed {affected} of {selected.Count} alert(s).\n\n" +
        $"{archived} alert(s) have been archived and cannot be dismissed from this view.",
        "Partial Dismiss",
        MessageBoxButton.OK,
        MessageBoxImage.Information);
}

This directly addresses the user's confusion in #718 — they click dismiss and "nothing happens" with no explanation.

---

7. Add structured telemetry to the dismiss path

Files: Lite/Services/LocalDataService.AlertHistory.cs

Add Stopwatch timing and diagnostic counters that always fire (not gated by LogAlertDismissals):

var sw = System.Diagnostics.Stopwatch.StartNew();
// ... existing dismiss logic ...
sw.Stop();
AppLogger.Info("AlertDismiss",
    $"Dismiss completed in {sw.ElapsedMilliseconds}ms: " +
    $"{totalAffected}/{alerts.Count} updated, " +
    $"connection_state={connection.State}");

This data directly answers whether the issue is lock contention (high elapsed time), archival timing (0 affected), or connection issues (unexpected state).

---

8. Detect and warn about in-progress archival

Files: Lite/Controls/AlertsHistoryTab.xaml.cs, Lite/Services/ArchiveService.cs

Expose the archive lock state via a static property:

// ArchiveService.cs
public static bool IsArchiving => s_archiveLock.CurrentCount == 0;

In the dismiss handlers, check before attempting:

if (ArchiveService.IsArchiving)
{
    MessageBox.Show(
        "Data archival is in progress. Please retry in a moment.",
        "Please Wait",
        MessageBoxButton.OK,
        MessageBoxImage.Information);
    return;
}

---

9. Make "Dismiss All" parquet-aware

Files: Lite/Services/LocalDataService.AlertHistory.cs

If suggestion #2 (sidecar table) is implemented, DismissAllVisibleAlertsAsync also needs to INSERT matching parquet-only rows into dismissed_archive_alerts. After the live-table UPDATE, run:

INSERT INTO dismissed_archive_alerts (alert_time, server_id, metric_name)
SELECT p.alert_time, p.server_id, p.metric_name
FROM read_parquet('{globPath}', union_by_name=true) p
WHERE p.alert_time >= $cutoff
AND   p.dismissed = FALSE
AND   ($serverId IS NULL OR p.server_id = $serverId)
AND   NOT EXISTS (
    SELECT 1 FROM dismissed_archive_alerts d
    WHERE d.alert_time = p.alert_time
    AND   d.server_id  = p.server_id
    AND   d.metric_name = p.metric_name
)

---

10. Add a post-dismiss verification query

Files: Lite/Services/LocalDataService.AlertHistory.cs, Lite/Controls/AlertsHistoryTab.xaml.cs

After DismissAlertsAsync completes, re-query the specific alerts from the view:

var survivors = new List<string>();
foreach (var alert in alerts)
{
    using var checkCmd = connection.CreateCommand();
    checkCmd.CommandText = @"
        SELECT COUNT(1) FROM v_config_alert_log
        WHERE alert_time = $1 AND server_id = $2
        AND metric_name = $3 AND dismissed = FALSE";
    // ... bind params ...
    var still_visible = (long)(await checkCmd.ExecuteScalarAsync())! > 0;
    if (still_visible)
        survivors.Add($"{alert.MetricName} @ {alert.ServerName} ({alert.AlertTime:HH:mm:ss})");
}

Return the survivors list to the caller so the UI can report exactly which alerts couldn't be dismissed and why. This turns a silent failure into an observable, debuggable event.

---

11. Batch UPDATEs into a single statement

Files: Lite/Services/LocalDataService.AlertHistory.cs

Instead of N individual UPDATE statements (one per selected alert), build a single statement:

UPDATE config_alert_log
SET    dismissed = TRUE
WHERE  dismissed = FALSE
AND    (alert_time, server_id, metric_name) IN (
    VALUES ($1, $2, $3), ($4, $5, $6), ...
)

This reduces N round-trips to 1, minimizes lock-hold duration, and returns a single affected-rows count that's easier to reason about.

---

12. Add a stale-data indicator to the grid

Files: Lite/Controls/AlertsHistoryTab.xaml.cs, Lite/Controls/AlertsHistoryTab.xaml

Track when the grid was last refreshed:

private DateTime _lastRefresh = DateTime.MinValue;

private async Task LoadAlertsAsync()
{
    // ... existing code ...
    _lastRefresh = DateTime.UtcNow;
}

Add a subtle TextBlock near the alert count indicator that shows when data was last loaded. If the data is older than the archival interval, change the text to a warning: "Data may be stale — click Refresh." This covers the edge case where the user leaves the alerts tab open, archival runs, and the grid shows alerts that no longer exist in the live table.

[HannahVernon]

@vpnicholls - the default archival period where alerts get moved into read-only parquet files is 7 days. Once they are archived to parquet files, they can no longer be dismissed in the current version of the product.

When you select "Last 7 Days" or "All" as the time range, alerts that have crossed the 7-day boundary show up in the grid (via parquet) but can't be dismissed. I'm assuming the alerts you're having trouble with are not yet 7 days old?

[erikdarlingdata]

@HannahVernon there are quite a few options in here, and I'm not entirely sure which will specifically address the mute/ack/silence gremlins best. I'll leave this in your capable hands.

[HannahVernon]

Since the logging options I just added via PR #721 may help the OP in this question, perhaps we should get them merged in, then I'll kick off a branch for the other items.

[vpnicholls]

@vpnicholls - the default archival period where alerts get moved into read-only parquet files is 7 days. Once they are archived to parquet files, they can no longer be dismissed in the current version of the product.

When you select "Last 7 Days" or "All" as the time range, alerts that have crossed the 7-day boundary show up in the grid (via parquet) but can't be dismissed. I'm assuming the alerts you're having trouble with are not yet 7 days old?

Yes, @HannahVernon , correct. It's for alerts that haven't yet crossed the 7-day threshold.

I don't get all the nitty gritty of the code in the suggested fixes, but I see from your descriptions what the underlying problem is. Thank you!

[HannahVernon]

Thanks for confirming! Once we have dismissal diagnostics in place we can hopefully nail down what is causing alerts to not be dismissed.

[vpnicholls]

Not sure if it's helpful or relevant, but I just noticed a couple of alerts that came up were in red rows. I could dismiss these. But the ones I can't dismiss are in orange rows.

[HannahVernon]

Curious, the colors should indicate the following:

Alert History Row Colors

Color	Condition	Meaning
🔴 Red tint	Metric contains "Deadlock" or "Poison"	Critical alerts — deadlocks and poison wait detections
🟠 Orange tint	Not resolved and not critical	Standard warnings — high CPU, blocking, long-running queries, TempDB space, etc.
🟢 Green tint	Metric contains "Cleared" or "Resolved"	Alert condition has resolved on its own
🔅 50% opacity, italic	Alert was muted	A mute rule matched this alert — the alert was still logged but no tray/email notification was sent

All background tints are at 20% opacity to keep the grid readable.

I'll continue to look into this issue. @vpnicholls any further points you notice would be helpful, thank you!