docs(research): 5 research artifacts + 3 bench harnesses from overnight fleet run#222
docs(research): 5 research artifacts + 3 bench harnesses from overnight fleet run#222Gradata wants to merge 1 commit into
Conversation
…ht fleet run Overnight 2026-05-20→21 the autonomous research fleet (analyst agent on claude-sonnet-4-6) produced these deliverables but heartbeat budgets ran out before agents pushed them to git. Surfaced today as uncommitted-but-real work in the SDK working tree. Promoting them manually so the work isn't lost. Research docs: - convergence-curve-math.md (206 lines): exponential / power law / smoothed-MA / cumulative-plateau comparison with shipping recommendation - patch-acceptance-2026-05-21.md (222): self-healing patch acceptance rate study - graduation-quality-2026-05-21.md (231): meta-rule graduation quality audit - many-shot-ablation-2026-05-21.md (244): k=10/20/50 ablation - embedding-vs-bm25-2026-05-21.md (283): cross-language scoring comparison Bench harnesses (runnable): - bench/curve_fitting.py (537): fits the 4 curve models, exports PNG charts + JSON results for the convergence research recommendation - bench/many_shot_ablation.py (628): bench harness for the many-shot ablation - bench/cross_language_scoring.py (630): bench for BM25 vs sentence-embedding scoring Authored: analyst agent (claude-sonnet-4-6) via Paperclip company fleet. Reviewed-by: parent agent (this PR) — content is substantive, references real codebase paths, recommendations have R²/AIC math behind them. Refs: research issues afeac9d4, b3e07178, 6cecf363, 4f527f65
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
Your free trial has ended. If you'd like to continue receiving code reviews, you can add a payment method here.
📝 WalkthroughSummary
WalkthroughThis PR adds three new benchmark harnesses ( ChangesBenchmark and Research Infrastructure
Estimated code review effort🎯 4 (Complex) | ⏱️ ~50 minutes Suggested labels
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 OpenGrep (1.21.0)OpenGrep fatal error (exit code 2): �[32m✔�[39m �[1mOpengrep OSS�[0m �[1m Loading rules from local config...�[0m Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 16
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@Gradata/bench/cross_language_scoring.py`:
- Line 425: The call a(f"**GRA-1299**: embedding vs BM25 on zero-term-overlap
probes ") uses an unnecessary f-string prefix causing Ruff F541; replace the
f-string with a plain string literal by removing the leading "f" in the argument
to function a (i.e., change the call in cross_language_scoring.py where
a(f"...") is used to a("...")), and verify there are no interpolations that
require f-strings before committing.
- Line 391: The p95 calculation uses int(0.95 * len(latencies)) which can
overshoot; change it to compute the 95th-rank as ceil(0.95 * n) - 1 and index
sorted(latencies) with that bounded index (and handle empty latencies). Update
the line computing p95_lat (referencing the latencies list and p95_lat variable)
to calculate rank = max(0, min(len(latencies)-1, math.ceil(0.95 *
len(latencies)) - 1)) and then use sorted(latencies)[rank]; ensure math.ceil is
imported/available and guard against empty latencies to avoid IndexError.
In `@Gradata/bench/curve_fitting.py`:
- Around line 272-280: When Matplotlib is unavailable make_charts currently
returns {} which breaks ProfileResult.chart_path and JSON output; change the
exception path to return a dict mapping the profile's model identifier to an
empty string (e.g., {profile.model: ""}) so the function always returns string
paths. Update the same pattern in the other try/except blocks referenced (around
the blocks at the later occurrences) so each returns a mapping with the
appropriate model key to an empty string instead of an empty dict; ensure
references are to make_charts and ProfileResult.chart_path so callers always get
a str path value.
- Around line 479-488: The current SQL filters out sessions with zero
corrections by using WHERE type = 'CORRECTION' and COUNT(*); update the query
used where db_path/conn are defined to compute per-session correction counts
including zeros: remove the type filter and replace COUNT(*) with SUM(CASE WHEN
type = 'CORRECTION' THEN 1 ELSE 0 END) AS cnt, keeping the session IS NOT NULL
AND session > 0 condition and the GROUP BY session ORDER BY session; make the
same change to the second occurrence around lines 491-495.
- Around line 132-137: The _r2 function currently returns 1.0 whenever ss_tot ==
0, which yields false perfect scores for constant y_true; change the logic in
_r2 to only return 1.0 when both ss_tot == 0 and ss_res == 0 (i.e., predictions
exactly match the constant target), otherwise return 0.0 for the constant-target
case so a non-matching prediction is not reported as perfect; update the branch
in _r2 that handles ss_tot == 0 accordingly and keep the existing 1.0 - ss_res /
ss_tot behavior for the general case.
In `@Gradata/bench/many_shot_ablation.py`:
- Around line 133-140: The math assumes k injected items even though top_k is
truncated to corpus length; update calculations to use the actual number of
retrieved/injected items (n = len(top_k)) instead of k when computing precision,
fp_count, and any downstream cost like context_tokens; specifically, replace
uses of k in the precision and fp_count formulas with n (and guard division by
zero), and apply the same change near the other occurrence (context_tokens /
related logic) so all noise/cost metrics reflect the true number of returned
items (use symbols ranked_indices, top_k, relevant_set, relevant_in_top_k,
precision, fp_count, context_tokens, and probe.relevant_indices to locate and
fix the code).
- Around line 153-164: Guard against empty probe sets by checking probe_results
and latencies before doing aggregations: if probe_results is empty (e.g.,
num_probes==0 or _build_probes returned []), avoid dividing by n and computing
p95; instead set n=0 and safe defaults (coverage_rate=0.0, mean_precision=0.0,
mean_fp_count=0.0, fp_rate=0.0, compliance_est=0.0) and for latencies set
avg_lat and p95_lat to None or 0.0; implement this check immediately before the
existing calculations that compute n, coverage_rate, mean_precision,
mean_fp_count, fp_rate, compliance_est, avg_lat and p95_lat, and ensure
NOISE_FACTOR usage remains guarded by the empty-check so you never divide by
zero or index into an empty sorted(latencies).
In `@Gradata/docs/research/embedding-vs-bm25-2026-05-21.md`:
- Around line 49-55: Update the construction rule wording to match the reported
outcomes: replace the strict "Jaccard token similarity < 0.05" requirement with
a relaxed/accurate statement (e.g., "Jaccard token similarity ≤ 0.07" or
"primarily < 0.05, with two exceptions at 0.06–0.07") so the rule and the
results (28 probes J=0.00; probes 11 and 12 at 0.06–0.07) are consistent; keep
reference to applying the same stopword list and tokenizer as jit_inject.py and
update the phrase that currently reads "the probe must have a Jaccard token
similarity < 0.05" accordingly.
In `@Gradata/docs/research/graduation-quality-2026-05-21.md`:
- Around line 47-53: The fenced code blocks in the markdown are missing a
language tag and lack surrounding blank lines; update each problematic fence
(the block shown and the ones starting at the other noted fences) to use a
language (e.g., ```text) and ensure there is a blank line immediately before and
after each fenced block so they pass MD040 and MD031. Also mirror this change
where similar examples are generated in the generator code path around the
_passes_beta_lb_gate() area in _graduation.py so future output includes the
```text fence and blank-line padding.
- Around line 1-2: The H1 heading "Graduation Quality Audit — GRA-1293" lacks a
trailing blank line; add a single blank line immediately after that heading so
there's an empty line between the H1 and the following metadata line to satisfy
MD022 (blanks-around-headings).
- Line 137: The sentence in REC-2's rationale ("Beta(α=4, β=1) at the 5th
percentile gives ~0.48 LB, which can exceed 0.75") is self-contradictory; update
the phrasing to a correct quantitative claim by replacing "which can exceed
0.75" with a correct relation (e.g., "which is well below 0.75" or specify the
correct percentile/value if you meant a different prior), and ensure the
surrounding sentence about requiring 5 observations instead of 3 is consistent
with the corrected numeric statement; look for the exact phrase "Beta(α=4, β=1)
at the 5th percentile gives ~0.48 LB" in the REC-2 rationale and edit that
sentence only.
In `@Gradata/docs/research/many-shot-ablation-2026-05-21.md`:
- Around line 40-42: Add a language tag to the fenced code block containing the
formula `compliance_est(k) = coverage(k) × (1 − 0.30 × fp_rate(k))` (e.g.,
change ``` to ```text) so markdownlint rule MD040 is satisfied; ensure the
opening fence contains the language token and the closing fence remains
unchanged.
In `@Gradata/docs/research/patch-acceptance-2026-05-21.md`:
- Line 6: Replace the stale branch identifier "GRA-1291-prompt-injection-survey"
in the document header with the correct branch name from the PR metadata
("docs/research-overnight-fleet-deliverables") so the header accurately reflects
the delivering branch; locate the header line containing the branch token and
update that string accordingly to maintain correct traceability.
- Around line 113-115: The fenced code block containing the snippet
"<original_rule> (especially in context: word1 word2 word3)" is missing a
language identifier; update that fenced block in the document so the opening
triple-backticks include a language (e.g., use "text") to satisfy markdownlint
MD040 and ensure consistent rendering—locate the fenced block that begins with
``` before the "<original_rule>" line and change it to ```text.
- Around line 176-191: The fenced JSON example (the block starting with ```json
and the shown rule_patch_observed object) lacks blank lines before and after the
code fence, violating MD031; fix it by inserting a blank line immediately above
the opening ```json fence and another blank line immediately below the closing
``` fence so the fenced code block is separated from surrounding text and
satisfies markdownlint.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 2d84943b-fd9e-450f-ac67-fd9e64bb87f1
📒 Files selected for processing (8)
Gradata/bench/cross_language_scoring.pyGradata/bench/curve_fitting.pyGradata/bench/many_shot_ablation.pyGradata/docs/research/convergence-curve-math.mdGradata/docs/research/embedding-vs-bm25-2026-05-21.mdGradata/docs/research/graduation-quality-2026-05-21.mdGradata/docs/research/many-shot-ablation-2026-05-21.mdGradata/docs/research/patch-acceptance-2026-05-21.md
📜 Review details
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (8)
- GitHub Check: pytest (py3.12)
- GitHub Check: pytest macos-latest / py3.12
- GitHub Check: pytest windows-latest / py3.12
- GitHub Check: pytest ubuntu-latest / py3.12
- GitHub Check: pytest (py3.11)
- GitHub Check: pytest macos-latest / py3.11
- GitHub Check: pytest windows-latest / py3.11
- GitHub Check: pytest ubuntu-latest / py3.11
🧰 Additional context used
🪛 LanguageTool
Gradata/docs/research/patch-acceptance-2026-05-21.md
[style] ~107-~107: Consider an alternative for the overused word “exactly”.
Context: ... behavioral filter is needed — which is exactly what _patches.py provides. ### 2. Th...
(EXACTLY_PRECISELY)
Gradata/docs/research/convergence-curve-math.md
[style] ~119-~119: Consider an alternative for the overused word “exactly”.
Context: ... both parametric models fail — which is exactly where Mann-Kendall (already implemented...
(EXACTLY_PRECISELY)
[style] ~143-~143: To form a complete sentence, be sure to include a subject.
Context: ...LS slope:** The current implementation. Should be replaced. Linear fit on a decaying s...
(MISSING_IT_THERE)
Gradata/docs/research/many-shot-ablation-2026-05-21.md
[uncategorized] ~117-~117: If this is a compound adjective that modifies the following noun, use a hyphen.
Context: ...eals a corpus gap, not a k problem. The two TONE probes that miss at k=50 are stylistica...
(EN_COMPOUND_ADJECTIVE_INTERNAL)
[style] ~149-~149: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...ctive on this corpus. k=20 breaks even. k=50 is near-global injection rather than...
(ENGLISH_WORD_REPEAT_BEGINNING_RULE)
Gradata/docs/research/embedding-vs-bm25-2026-05-21.md
[style] ~82-~82: If ‘chance’ means ‘possibility’, this phrase is redundant. Consider writing “chance”.
Context: ...33** | 0.714 | 0.833 | 14.40 | Random chance on a 30-document corpus is P@1 = 0.033 ...
(RANDOM_CHANCE)
🪛 markdownlint-cli2 (0.22.1)
Gradata/docs/research/patch-acceptance-2026-05-21.md
[warning] 113-113: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
[warning] 177-177: Fenced code blocks should be surrounded by blank lines
(MD031, blanks-around-fences)
Gradata/docs/research/many-shot-ablation-2026-05-21.md
[warning] 40-40: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
Gradata/docs/research/graduation-quality-2026-05-21.md
[warning] 1-1: Headings should be surrounded by blank lines
Expected: 1; Actual: 0; Below
(MD022, blanks-around-headings)
[warning] 47-47: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
[warning] 109-109: Fenced code blocks should be surrounded by blank lines
(MD031, blanks-around-fences)
[warning] 129-129: Fenced code blocks should be surrounded by blank lines
(MD031, blanks-around-fences)
[warning] 151-151: Fenced code blocks should be surrounded by blank lines
(MD031, blanks-around-fences)
[warning] 182-182: Fenced code blocks should be surrounded by blank lines
(MD031, blanks-around-fences)
🪛 Ruff (0.15.13)
Gradata/bench/curve_fitting.py
[warning] 104-107: Use ternary operator base = 5.0 * math.exp(-0.15 * n) + 1.0 if n <= 20 else 1.0 + rng.gauss(0, 0.3) instead of if-else-block
Replace if-else-block with base = 5.0 * math.exp(-0.15 * n) + 1.0 if n <= 20 else 1.0 + rng.gauss(0, 0.3)
(SIM108)
Gradata/bench/many_shot_ablation.py
[error] 353-353: f-string without any placeholders
Remove extraneous f prefix
(F541)
[error] 363-363: f-string without any placeholders
Remove extraneous f prefix
(F541)
[error] 373-373: f-string without any placeholders
Remove extraneous f prefix
(F541)
[error] 394-394: f-string without any placeholders
Remove extraneous f prefix
(F541)
[error] 608-608: f-string without any placeholders
Remove extraneous f prefix
(F541)
Gradata/bench/cross_language_scoring.py
[warning] 35-35: Import from collections.abc instead: Callable
Import from collections.abc
(UP035)
[error] 425-425: f-string without any placeholders
Remove extraneous f prefix
(F541)
🔇 Additional comments (2)
Gradata/docs/research/convergence-curve-math.md (1)
1-207: LGTM!Gradata/bench/many_shot_ablation.py (1)
353-353: ⚡ Quick winProvide the full original review comment and any verification outputs (shell/web results) to rewrite it
I don’t have the<review_comment>content or the verification results needed to produce an updated, accurate rewritten comment.
| } | ||
|
|
||
| avg_lat = sum(latencies) / len(latencies) | ||
| p95_lat = sorted(latencies)[int(0.95 * len(latencies))] |
There was a problem hiding this comment.
Fix p95 latency percentile indexing.
Current index selection can over-shoot the intended 95th-percentile rank for common sample sizes, so reported p95 can be inaccurate.
Proposed fix
- p95_lat = sorted(latencies)[int(0.95 * len(latencies))]
+ sorted_lat = sorted(latencies)
+ p95_idx = max(0, math.ceil(0.95 * len(sorted_lat)) - 1)
+ p95_lat = sorted_lat[p95_idx]🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@Gradata/bench/cross_language_scoring.py` at line 391, The p95 calculation
uses int(0.95 * len(latencies)) which can overshoot; change it to compute the
95th-rank as ceil(0.95 * n) - 1 and index sorted(latencies) with that bounded
index (and handle empty latencies). Update the line computing p95_lat
(referencing the latencies list and p95_lat variable) to calculate rank = max(0,
min(len(latencies)-1, math.ceil(0.95 * len(latencies)) - 1)) and then use
sorted(latencies)[rank]; ensure math.ceil is imported/available and guard
against empty latencies to avoid IndexError.
|
|
||
| a(f"# cross-language-scoring benchmark — {run_date}") | ||
| a("") | ||
| a(f"**GRA-1299**: embedding vs BM25 on zero-term-overlap probes ") |
There was a problem hiding this comment.
Remove the unused f-string prefix to satisfy lint.
This is flagged as Ruff F541 and can block CI if lint errors are enforced.
Proposed fix
- a(f"**GRA-1299**: embedding vs BM25 on zero-term-overlap probes ")
+ a("**GRA-1299**: embedding vs BM25 on zero-term-overlap probes ")📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| a(f"**GRA-1299**: embedding vs BM25 on zero-term-overlap probes ") | |
| a("**GRA-1299**: embedding vs BM25 on zero-term-overlap probes ") |
🧰 Tools
🪛 Ruff (0.15.13)
[error] 425-425: f-string without any placeholders
Remove extraneous f prefix
(F541)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@Gradata/bench/cross_language_scoring.py` at line 425, The call
a(f"**GRA-1299**: embedding vs BM25 on zero-term-overlap probes ") uses an
unnecessary f-string prefix causing Ruff F541; replace the f-string with a plain
string literal by removing the leading "f" in the argument to function a (i.e.,
change the call in cross_language_scoring.py where a(f"...") is used to
a("...")), and verify there are no interpolations that require f-strings before
committing.
| def _r2(y_true: np.ndarray, y_pred: np.ndarray) -> float: | ||
| ss_res = float(np.sum((y_true - y_pred) ** 2)) | ||
| ss_tot = float(np.sum((y_true - np.mean(y_true)) ** 2)) | ||
| if ss_tot == 0: | ||
| return 1.0 | ||
| return 1.0 - ss_res / ss_tot |
There was a problem hiding this comment.
Handle constant-series R² without false perfect scores.
For constant y_true, returning 1.0 unconditionally can misreport a bad fit as perfect.
Proposed fix
def _r2(y_true: np.ndarray, y_pred: np.ndarray) -> float:
ss_res = float(np.sum((y_true - y_pred) ** 2))
ss_tot = float(np.sum((y_true - np.mean(y_true)) ** 2))
if ss_tot == 0:
- return 1.0
+ return 1.0 if ss_res == 0 else 0.0
return 1.0 - ss_res / ss_tot🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@Gradata/bench/curve_fitting.py` around lines 132 - 137, The _r2 function
currently returns 1.0 whenever ss_tot == 0, which yields false perfect scores
for constant y_true; change the logic in _r2 to only return 1.0 when both ss_tot
== 0 and ss_res == 0 (i.e., predictions exactly match the constant target),
otherwise return 0.0 for the constant-target case so a non-matching prediction
is not reported as perfect; update the branch in _r2 that handles ss_tot == 0
accordingly and keep the existing 1.0 - ss_res / ss_tot behavior for the general
case.
| def make_charts(profile: ProfileResult, out_dir: Path) -> dict[str, str]: | ||
| """Generate one multi-panel chart for this profile. Returns {model: path}.""" | ||
| try: | ||
| import matplotlib | ||
|
|
||
| matplotlib.use("Agg") | ||
| import matplotlib.pyplot as plt | ||
| except ImportError: | ||
| return {} |
There was a problem hiding this comment.
Fix make_charts return contract to always be a string path.
make_charts currently returns {} when Matplotlib is unavailable, which propagates a non-string into ProfileResult.chart_path and JSON output.
Proposed fix
-def make_charts(profile: ProfileResult, out_dir: Path) -> dict[str, str]:
- """Generate one multi-panel chart for this profile. Returns {model: path}."""
+def make_charts(profile: ProfileResult, out_dir: Path) -> str:
+ """Generate one multi-panel chart for this profile. Returns chart path or empty string."""
@@
- except ImportError:
- return {}
+ except ImportError:
+ return ""
@@
- return str(chart_path)
+ return str(chart_path)Also applies to: 366-367, 412-413
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@Gradata/bench/curve_fitting.py` around lines 272 - 280, When Matplotlib is
unavailable make_charts currently returns {} which breaks
ProfileResult.chart_path and JSON output; change the exception path to return a
dict mapping the profile's model identifier to an empty string (e.g.,
{profile.model: ""}) so the function always returns string paths. Update the
same pattern in the other try/except blocks referenced (around the blocks at the
later occurrences) so each returns a mapping with the appropriate model key to
an empty string instead of an empty dict; ensure references are to make_charts
and ProfileResult.chart_path so callers always get a str path value.
| import sqlite3 | ||
|
|
||
| db_path = Path(args.brain_path) / "system.db" | ||
| if db_path.exists(): | ||
| conn = sqlite3.connect(str(db_path)) | ||
| rows = conn.execute( | ||
| "SELECT session, COUNT(*) as cnt FROM events " | ||
| "WHERE type = 'CORRECTION' AND session IS NOT NULL AND session > 0 " | ||
| "GROUP BY session ORDER BY session" | ||
| ).fetchall() |
There was a problem hiding this comment.
Include zero-correction sessions when loading real brain data.
This query drops sessions with zero corrections, which biases the real profile and can change model ranking/recommendation.
Proposed fix
- rows = conn.execute(
- "SELECT session, COUNT(*) as cnt FROM events "
- "WHERE type = 'CORRECTION' AND session IS NOT NULL AND session > 0 "
- "GROUP BY session ORDER BY session"
- ).fetchall()
+ rows = conn.execute(
+ "SELECT session, "
+ "SUM(CASE WHEN type = 'CORRECTION' THEN 1 ELSE 0 END) AS cnt "
+ "FROM events "
+ "WHERE session IS NOT NULL AND session > 0 "
+ "GROUP BY session ORDER BY session"
+ ).fetchall()
@@
- real_sessions = [r[0] for r in rows]
- real_corrections = [float(r[1]) for r in rows]
+ real_sessions = [int(r[0]) for r in rows]
+ real_corrections = [float(r[1]) for r in rows]Also applies to: 491-495
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@Gradata/bench/curve_fitting.py` around lines 479 - 488, The current SQL
filters out sessions with zero corrections by using WHERE type = 'CORRECTION'
and COUNT(*); update the query used where db_path/conn are defined to compute
per-session correction counts including zeros: remove the type filter and
replace COUNT(*) with SUM(CASE WHEN type = 'CORRECTION' THEN 1 ELSE 0 END) AS
cnt, keeping the session IS NOT NULL AND session > 0 condition and the GROUP BY
session ORDER BY session; make the same change to the second occurrence around
lines 491-495.
| ``` | ||
| compliance_est(k) = coverage(k) × (1 − 0.30 × fp_rate(k)) | ||
| ``` |
There was a problem hiding this comment.
Add a language tag to the fenced code block (MD040).
Use a language like text for the formula block to satisfy markdownlint.
🧰 Tools
🪛 markdownlint-cli2 (0.22.1)
[warning] 40-40: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@Gradata/docs/research/many-shot-ablation-2026-05-21.md` around lines 40 - 42,
Add a language tag to the fenced code block containing the formula
`compliance_est(k) = coverage(k) × (1 − 0.30 × fp_rate(k))` (e.g., change ``` to
```text) so markdownlint rule MD040 is satisfied; ensure the opening fence
contains the language token and the closing fence remains unchanged.
| | k jump | Δ coverage | Δ compliance | Δ context tokens | Compliance / 100 extra tokens | | ||
| |--------|-----------|-------------|-----------------|-------------------------------| | ||
| | 5→10 | +0.025 | **−0.005** | +75 | −0.007 | | ||
| | 10→20 | +0.025 | +0.006 | +150 | +0.004 | | ||
| | 20→50 | +0.050 | +0.027 | +450 | +0.006 | |
There was a problem hiding this comment.
Align marginal-efficiency metric with harness output for reproducibility.
This table is compliance-per-100-tokens, but the harness/report generator currently emits coverage-per-100-tokens. With the raw output link on Line 240, readers won’t be able to reproduce this section as-is.
Also applies to: 240-240
| **Status:** INSTRUMENTED — telemetry live, behavioral data pending | ||
| **Date:** 2026-05-21 | ||
| **Author:** analyst (claude_local / sonnet-4-6) | ||
| **Branch:** GRA-1291-prompt-injection-survey |
There was a problem hiding this comment.
Fix stale branch metadata in the document header.
Line 6 lists GRA-1291-prompt-injection-survey, but this artifact is being delivered from docs/research-overnight-fleet-deliverables per PR metadata. This can mislead traceability for future audits.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@Gradata/docs/research/patch-acceptance-2026-05-21.md` at line 6, Replace the
stale branch identifier "GRA-1291-prompt-injection-survey" in the document
header with the correct branch name from the PR metadata
("docs/research-overnight-fleet-deliverables") so the header accurately reflects
the delivering branch; locate the header line containing the branch token and
update that string accordingly to maintain correct traceability.
| ``` | ||
| <original_rule> (especially in context: word1 word2 word3) | ||
| ``` |
There was a problem hiding this comment.
Add a language identifier to the fenced code block.
Line 113 starts a fenced block without a language tag (markdownlint MD040). Please label it (for example, text) to keep linting and rendering consistent.
Proposed fix
-```
+```text
<original_rule> (especially in context: word1 word2 word3)</details>
<!-- suggestion_start -->
<details>
<summary>📝 Committable suggestion</summary>
> ‼️ **IMPORTANT**
> Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
```suggestion
🧰 Tools
🪛 markdownlint-cli2 (0.22.1)
[warning] 113-113: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@Gradata/docs/research/patch-acceptance-2026-05-21.md` around lines 113 - 115,
The fenced code block containing the snippet "<original_rule> (especially in
context: word1 word2 word3)" is missing a language identifier; update that
fenced block in the document so the opening triple-backticks include a language
(e.g., use "text") to satisfy markdownlint MD040 and ensure consistent
rendering—locate the fenced block that begins with ``` before the
"<original_rule>" line and change it to ```text.
| Expected event schema: | ||
| ```json | ||
| { | ||
| "type": "rule_patch_observed", | ||
| "source": "_patches.observe_patch", | ||
| "data": { | ||
| "category": "TONE", | ||
| "old_rule_text": "Never use exclamation marks", | ||
| "new_rule_text": "Never use exclamation marks (especially in context: email removed draft)", | ||
| "applied_at": "2026-05-21T12:00:00+00:00", | ||
| "observed_compliance_before": 3, | ||
| "observed_compliance_after_3_sessions": null | ||
| }, | ||
| "tags": ["category:TONE", "self_healing", "patch_telemetry"] | ||
| } | ||
| ``` |
There was a problem hiding this comment.
Surround the JSON fence with blank lines to satisfy markdownlint.
The fenced JSON example around Line 177 should be separated by blank lines (MD031), which improves markdown parser compatibility.
Proposed fix
Expected event schema:
+
```json
{
"type": "rule_patch_observed",
"source": "_patches.observe_patch",
"data": {
"category": "TONE",
"old_rule_text": "Never use exclamation marks",
"new_rule_text": "Never use exclamation marks (especially in context: email removed draft)",
"applied_at": "2026-05-21T12:00:00+00:00",
"observed_compliance_before": 3,
"observed_compliance_after_3_sessions": null
},
"tags": ["category:TONE", "self_healing", "patch_telemetry"]
}</details>
<!-- suggestion_start -->
<details>
<summary>📝 Committable suggestion</summary>
> ‼️ **IMPORTANT**
> Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
```suggestion
Expected event schema:
🧰 Tools
🪛 markdownlint-cli2 (0.22.1)
[warning] 177-177: Fenced code blocks should be surrounded by blank lines
(MD031, blanks-around-fences)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@Gradata/docs/research/patch-acceptance-2026-05-21.md` around lines 176 - 191,
The fenced JSON example (the block starting with ```json and the shown
rule_patch_observed object) lacks blank lines before and after the code fence,
violating MD031; fix it by inserting a blank line immediately above the opening
```json fence and another blank line immediately below the closing ``` fence so
the fenced code block is separated from surrounding text and satisfies
markdownlint.
1 participant
Summary
Overnight 2026-05-20→21 the autonomous research fleet produced these deliverables but heartbeat budgets ran out before agents called
gh pr create. The files sat uncommitted on local disk all morning. Promoting them manually so the work isn't lost.Research docs (1,186 lines)
convergence-curve-math.md— 4-model comparison (exponential / power-law / smoothed-MA / cumulative-plateau) with shipping recommendationpatch-acceptance-2026-05-21.md— self-healing patch acceptance rate studygraduation-quality-2026-05-21.md— meta-rule graduation quality auditmany-shot-ablation-2026-05-21.md— k=10/20/50 ablationembedding-vs-bm25-2026-05-21.md— cross-language scoring comparisonBench harnesses (1,795 lines, runnable)
bench/curve_fitting.py— fits 4 curve models, exports PNG + JSONbench/many_shot_ablation.py— many-shot benchbench/cross_language_scoring.py— BM25 vs embedding benchReview focus
Content is substantive (not LLM slop). References real codebase paths, recommendations have R²/AIC math. Skim the convergence-curve doc first — that's the highest-leverage one and directly informs in-flight ENG issues [441311ff] (smoothed cumulative curve) and [029731fe] (exponential-fit overlay).
Out of scope
Implementation of the recommendations — those are separate ENG PRs in flight.