FEAT: CBT-Bench Dataset

[warisgill]

Integrate the CBT-Bench psychotherapy benchmark dataset from HuggingFace
(Psychotherapy-LLM/CBT-Bench) into PyRIT.

Changes

• New file: pyrit/datasets/seed_datasets/remote/cbt_bench_dataset.py — _CBTBenchDataset loader
• Modified: pyrit/datasets/seed_datasets/remote/__init__.py — registered the new loader
• New file: tests/unit/datasets/test_cbt_bench_dataset.py — 7 unit tests

Key Design Decisions

• Prompt value combines situation + thoughts (per reviewer feedback on FEAT: CBT-Bench Dataset #888)
• Harm categories set to ["psycho-social harms"] (per @jbolor21 and @romanlutz discussion on FEAT: CBT-Bench Dataset #888)
• Default config is core_fine_seed but supports all 39 HuggingFace subsets via config parameter
• core_belief_fine_grained stored in metadata for downstream use

Closes #865 (supersedes stale PR #888)

@romanlutz

Tests and Documentation

• Tests: 7 new unit tests added in tests/unit/datasets/test_cbt_bench_dataset.py. All tests mock _fetch_from_huggingface (no network calls). All 98
  dataset unit tests pass (91 existing + 7 new). Ruff lint clean.
• Documentation: No documentation changes needed. Dataset classes are internal (_ prefixed) and auto-register via SeedDatasetProvider.__init_subclass__.
  api.rst only lists SeedDatasetProvider, not individual dataset loaders.
• JupyText: Not applicable — no notebook or API documentation changes.

[warisgill]

@microsoft-github-policy-service agree company="Microsoft"

[Copilot]

Pull request overview

This pull request integrates the CBT-Bench (Cognitive Behavioral Therapy benchmark) dataset from HuggingFace into PyRIT, enabling evaluation of LLM safety and alignment in psychotherapy contexts. The implementation supersedes stale PR #888 and addresses issue #865.

Changes:

• Added a new remote dataset loader for CBT-Bench with support for 39 HuggingFace subsets
• Registered the new loader in the remote datasets init file
• Added comprehensive unit tests with 7 test cases covering initialization, fetching, edge cases, and metadata validation

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
pyrit/datasets/seed_datasets/remote/cbt_bench_dataset.py	New dataset loader class implementing fetch logic, combining situation and thoughts into prompt value, storing core beliefs in metadata
pyrit/datasets/seed_datasets/remote/__init__.py	Registered _CBTBenchDataset in imports and all list following alphabetical ordering
tests/unit/datasets/test_cbt_bench_dataset.py	Unit tests with fixtures and mocking covering normal operation, custom configs, edge cases, and metadata validation

Comments suppressed due to low confidence (1)

pyrit/datasets/seed_datasets/remote/cbt_bench_dataset.py:117

• The metadata field for core_belief_fine_grained is being set to a list, but the Seed.metadata field is typed as dict[str, Union[str, int]]. This creates a type mismatch. To fix this, either convert the list to a string (e.g., JSON string or comma-separated) before storing in metadata, or use a local variable annotation like other datasets do (from typing import Any; metadata: dict[str, Any] = {...}).

                metadata["core_belief_fine_grained"] = core_beliefs

[romanlutz]

Looks good! Did you run the integration test that fetches the dataset?

I also want @jbolor21 to take a look as this was their feature request.

[romanlutz]

Oh and we need to rerun the notebook that lists all the datasets. It's the one in doc/code/datasets/ with index 0 I think. It should add CBT bench as a new line after execution.

[warisgill]

@romanlutz Thank you for the feedback! I've addressed both items:

Ready for review!

[anandansundar]

Why do we have personal workspace path here?

[romanlutz]

If you run in devcontainer it won't show but at the end of the day it does not matter