FEAT Add supports_multi_turn property to targets and adapt attacks accordingly

[romanlutz]

Problem

Some targets (e.g., OpenAIImageTarget, OpenAIVideoTarget) are fundamentally single-turn — they process one prompt at a time and don't use conversation
history. However, multi-turn attacks like RedTeamingAttack reuse the same conversation_id across turns, which causes failures when targets validate that
no prior messages exist in a conversation.

There was no formal mechanism for targets to declare single vs. multi-turn support, and no way for attacks to adapt their behavior accordingly.

Solution

1. Target property — Add a supports_multi_turn property to the target hierarchy:

• PromptTarget → False (default for stateless targets)
• PromptChatTarget → True (chat targets maintain conversation state)
• Explicit False overrides on single-turn OpenAI targets: OpenAIImageTarget, OpenAIVideoTarget, OpenAITTSTarget, OpenAICompletionTarget
• Explicit True overrides on stateful non-chat targets: RealtimeTarget, PlaywrightTarget, PlaywrightCopilotTarget, WebSocketCopilotTarget
• Conversation-length validation in OpenAIImageTarget and OpenAIVideoTarget _validate_request as a safety net

2. Attack adaptations:

• RedTeamingAttack: Rotates conversation_id before each turn for single-turn targets, tracking prior conversations as ConversationType.PRUNED in
  related_conversations
• TAP: Skips conversation history duplication for single-turn targets, creating fresh conversations per tree node
• CrescendoAttack, ChunkedRequestAttack, MultiPromptSendingAttack: Raise ValueError in _setup_async when used with single-turn targets (fundamentally
  incompatible — these attacks rely on building up conversation context)

3. Rotation helper — _rotate_conversation_for_single_turn_target on MultiTurnAttackStrategy base class:

• No-op for multi-turn targets and on the first turn (executed_turns == 0)
• Generates a new conversation_id and records the old one as ConversationType.PRUNED

Testing

• 9 unit tests for supports_multi_turn property values across the target hierarchy
• 7 unit tests for attack behaviors (rotation helper no-ops, rotation activation, ValueError guards)
• All 4649 existing unit tests continue to pass
• End-to-end verified: RedTeamingAttack with OpenAIImageTarget ran 2 turns successfully with conversation rotation

Related

• Complements FEAT backend attack API - revisiting attacks & conversations #1419 which restricts image/video targets to single-turn via _validate_request; this PR adds the proactive mechanism for attacks to adapt

[Copilot]

Pull request overview

This PR introduces a first-class capability flag (supports_multi_turn) on prompt targets so multi-turn attacks can adapt their conversation handling when interacting with fundamentally single-turn targets (e.g., image/video/TTS/completions).

Changes:

• Add supports_multi_turn to the prompt target hierarchy (default False, True for chat targets, with explicit overrides for specific targets).
• Adapt multi-turn attacks to rotate or avoid conversation history for single-turn targets, and add guards for attacks that require multi-turn state.
• Add unit tests covering target capability values and attack behavior/guards.

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/unit/target/test_supports_multi_turn.py	Verifies supports_multi_turn values across target classes.
tests/unit/executor/attack/multi_turn/test_supports_multi_turn_attacks.py	Tests conversation rotation helper and single-turn incompatibility guards.
pyrit/prompt_target/common/prompt_target.py	Adds supports_multi_turn default property on base target.
pyrit/prompt_target/common/prompt_chat_target.py	Overrides supports_multi_turn=True for chat targets.
pyrit/prompt_target/openai/openai_image_target.py	Marks image target single-turn; adds conversation-length safety check.
pyrit/prompt_target/openai/openai_video_target.py	Marks video target single-turn; adds conversation-length safety check.
pyrit/prompt_target/openai/openai_tts_target.py	Marks TTS target single-turn.
pyrit/prompt_target/openai/openai_completion_target.py	Marks completions target single-turn.
pyrit/prompt_target/openai/openai_realtime_target.py	Marks realtime target as multi-turn capable.
pyrit/prompt_target/playwright_target.py	Marks Playwright target as multi-turn capable.
pyrit/prompt_target/playwright_copilot_target.py	Marks Playwright Copilot target as multi-turn capable.
pyrit/prompt_target/websocket_copilot_target.py	Marks WebSocket Copilot target as multi-turn capable.
pyrit/executor/attack/multi_turn/multi_turn_attack_strategy.py	Adds _rotate_conversation_for_single_turn_target helper.
pyrit/executor/attack/multi_turn/red_teaming.py	Rotates conversation_id per turn for single-turn targets.
pyrit/executor/attack/multi_turn/tree_of_attacks.py	Avoids history duplication for single-turn targets (fresh conversation_id).
pyrit/executor/attack/multi_turn/multi_prompt_sending.py	Raises on single-turn targets in _setup_async.
pyrit/executor/attack/multi_turn/crescendo.py	Raises on single-turn targets in _setup_async.
pyrit/executor/attack/multi_turn/chunked_request.py	Raises on single-turn targets in _setup_async.

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.

[Copilot]

The new rotation helper logs via the module-level logger rather than the strategy’s injected logger (self._logger). This means callers providing a custom logger won’t see these messages (and it breaks consistency with other AttackStrategy logging). Use self._logger.debug(...) here instead of the global logger.

Suggested change

	logger.debug(
	self._logger.debug(

[Copilot]

request = message.message_pieces[0] is inconsistent with the rest of the method (which already extracts text_piece) and needlessly relies on piece ordering. Prefer using text_piece.conversation_id (or message.conversation_id) when checking for prior conversation history so this validation stays correct even if piece ordering changes.

Suggested change

	request = message.message_pieces[0]
	messages = self._memory.get_conversation(conversation_id=request.conversation_id)
	messages = self._memory.get_conversation(conversation_id=text_piece.conversation_id)

[rlundeen2]

One of the big problems I have with this, is that it really depends. Does the playwright target support multi-turn? Really depends on the implementation. It may not remember history.

[rlundeen2]

The same issue is present for HTTPTarget, likely AMLChatTarget.

Is there a way we could address it? Like make it a settable property vs a class property?

[rlundeen2]

legitimate to say never. Crescendo without backtracking is just RTO

[rlundeen2]

It would be nice to be able to get several properties about what a target is. It'd be nice to combine them. I also don't think they can be static per class in all cases (see my comments below about weird exceptions).

A proposal is we might have something like a TargetCapabilities class.

@dataclass
class TargetCapabilities:
    """
    Describes the capabilities of a PromptTarget so that orchestrators
    and other components can adapt their behavior accordingly.
    """

    # Whether the target natively supports multi-turn conversations
    # (i.e., it accepts and uses conversation_id / sequence tracking).
    multi_turn_support: bool = False

    # Whether previously sent messages in a conversation can be edited
    # or rewritten before sending the next turn.
    editable_history: bool = False

    # Whether the target supports requesting output conforming to a
    # JSON schema (e.g., OpenAI's response_format with json_schema).
    json_schema_support: bool = False

    # Whether the target supports a basic JSON output mode
    # (e.g., response_format={"type": "json_object"}) without
    # requiring a full schema.
    json_output_support: bool = False

    # Whether the target supports system / developer messages.
    system_message_support: bool = False

    # Input modalities for the target
    input_modality_support: list[PromptDataType]

    # Output modalities for the target
    output_modality_support: list[PromptDataType]

A class can have defaults here, but I think users should be able to overwrite these in many cases (perhaps as constructor arguments). E.g. if you have a Playwright target, it may or may not support many of these features. So I would say it's not static per class, but is static per instance.

So with a PromptTarget you could get_capabilities() and you get this object.

WDYT?

[rlundeen2]

One other thought, some of these are supported with metadata keys. So it might be a good time to think through how to connect that with this so we have the metadata keys for targets in one place. Not necessarily needed now.

[romanlutz]

Adding this for just multi turn now, and the rest in separate PRs. I love the idea.

[Copilot]

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 1 comment.

[Copilot]

_rotate_conversation_for_single_turn_target relies on an inline from pyrit.memory import CentralMemory import. This module doesn't appear to have a circular dependency with pyrit.memory, so the import should be moved to the top of the file to match the project's import-organization convention and avoid repeated imports on every rotation call.

[Copilot]

Pull request overview

Copilot reviewed 22 out of 23 changed files in this pull request and generated 3 comments.

[Copilot]

OpenAITarget.__init__ now uses a local import (from pyrit.auth import get_azure_openai_auth). This doesn’t appear to be needed to avoid a circular dependency (pyrit.auth doesn’t import OpenAITarget), and it violates the project’s “imports at top of file” convention. Please move the import to the module imports, or add an explicit comment explaining the circular dependency if there is one.

[Copilot]

The Azure-endpoint detection for Entra fallback is currently "azure" in endpoint_value.lower(). This heuristic is overly broad (can match non-OpenAI Azure-hosted domains) and also too narrow (would miss Azure OpenAI behind a custom domain). Consider parsing the URL and checking for known Azure OpenAI / Foundry host patterns (e.g., .openai.azure.com, .models.ai.azure.com) instead of a substring match.

[Copilot]

PromptTarget.__init__ still allows positional arguments even though it has multiple parameters, and this PR adds another (capabilities). Most other constructors in this area are keyword-only (def __init__(self, *, ...)), which reduces accidental argument ordering bugs. Consider making this initializer keyword-only (possibly via a deprecation period) to keep the base target API consistent and safer to extend.

[Copilot]

Pull request overview

Copilot reviewed 22 out of 25 changed files in this pull request and generated 5 comments.

[Copilot]

get_azure_openai_auth is imported inside __init__. Unless this is required to break a circular dependency, prefer a top-level import to keep imports consistent and avoid repeated import work. If the local import is intentional for circularity, add a brief comment explaining that rationale.

[Copilot]

This notebook output includes local user paths (e.g., C:\\Users\\romanlutz\\...), which is sensitive/PII and makes the docs non-portable. Please clear notebook outputs and re-run the notebook sanitization hooks so paths are removed before committing.

[Copilot]

The committed notebook output embeds a long, step-by-step harmful instruction payload inside an error log. This should not be checked into the repository (it’s both unsafe content and creates huge diffs). Please clear outputs (or at least remove/redact this output) before committing the notebook.

[Copilot]

patch_central_database is applied at the class level via @pytest.mark.usefixtures, but it is also injected as an unused function argument in these tests. Consider removing the unused parameter from the test signatures to avoid confusion and keep the tests minimal.

[Copilot]

The capabilities setter mutates behavior-affecting state, but Identifiable.get_identifier() caches identifiers for the object lifetime. If get_identifier() was called before capabilities is reassigned, the identifier will permanently reflect the old supports_multi_turn value. Consider either (a) making capabilities immutable after construction, or (b) resetting self._identifier to None in the setter so the identifier can be rebuilt with the updated capabilities.

Suggested change

	self._capabilities = value
	self._capabilities = value
	# Invalidate cached identifier so it can be rebuilt with updated capabilities.
	self._identifier = None

[Copilot]

Pull request overview

Copilot reviewed 23 out of 27 changed files in this pull request and generated 4 comments.

[Copilot]

AzureContentFilterScorer.__init__ advertises api_key can be Callable[[], str | Awaitable[str]], but the implementation and TokenProviderCredential treat callables as synchronous (no awaiting). Passing an async token provider would yield a coroutine object string and break auth. Either narrow the accepted type/docstring to sync callables only, or add wrapping logic similar to _ensure_async_token_provider.

[Copilot]

The newly committed notebook outputs include absolute local file paths (e.g., C:\\Users\\..., local repo paths) in tracebacks/logs. These leak developer machine details and tend to make docs non-reproducible; please clear/redact outputs before committing.

Suggested change

	"\u001b[37m File \"C:\\Users\\romanlutz\\git\\PyRIT4\\pyrit\\prompt_normalizer\\prompt_normalizer.py\", line 107, in\u001b[0m\n",
	"\u001b[37m send_prompt_async\u001b[0m\n",
	"\u001b[37m responses = await target.send_prompt_async(message=request)\u001b[0m\n",
	"\u001b[37m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\u001b[0m\n",
	"\u001b[37m File \"C:\\Users\\romanlutz\\git\\PyRIT4\\pyrit\\prompt_target\\common\\utils.py\", line 57, in\u001b[0m\n",
	"\u001b[37m set_max_rpm\u001b[0m\n",
	"\u001b[37m return await func(*args, **kwargs)\u001b[0m\n",
	"\u001b[37m ^^^^^^^^^^^^^^^^^^^^^^^^^^^\u001b[0m\n",
	"\u001b[37m File \"C:\\Users\\romanlutz\\AppData\\Local\\anaconda3\\Lib\\site-\u001b[0m\n",
	"\u001b[37m packages\\tenacity\\asyncio\\__init__.py\", line 193, in async_wrapped\u001b[0m\n",
	"\u001b[37m return await copy(fn, *args, **kwargs)\u001b[0m\n",
	"\u001b[37m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\u001b[0m\n",
	"\u001b[37m File \"C:\\Users\\romanlutz\\AppData\\Local\\anaconda3\\Lib\\site-\u001b[0m\n",
	"\u001b[37m packages\\tenacity\\asyncio\\__init__.py\", line 112, in __call__\u001b[0m\n",
	"\u001b[37m do = await self.iter(retry_state=retry_state)\u001b[0m\n",
	"\u001b[37m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\u001b[0m\n",
	"\u001b[37m File \"C:\\Users\\romanlutz\\AppData\\Local\\anaconda3\\Lib\\site-\u001b[0m\n",
	"\u001b[37m File \"pyrit\\prompt_normalizer\\prompt_normalizer.py\", line 107, in\u001b[0m\n",
	"\u001b[37m send_prompt_async\u001b[0m\n",
	"\u001b[37m responses = await target.send_prompt_async(message=request)\u001b[0m\n",
	"\u001b[37m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\u001b[0m\n",
	"\u001b[37m File \"pyrit\\prompt_target\\common\\utils.py\", line 57, in\u001b[0m\n",
	"\u001b[37m set_max_rpm\u001b[0m\n",
	"\u001b[37m return await func(*args, **kwargs)\u001b[0m\n",
	"\u001b[37m ^^^^^^^^^^^^^^^^^^^^^^^^^^^\u001b[0m\n",
	"\u001b[37m File \"tenacity\\asyncio\\__init__.py\", line 193, in async_wrapped\u001b[0m\n",
	"\u001b[37m return await copy(fn, *args, **kwargs)\u001b[0m\n",
	"\u001b[37m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\u001b[0m\n",
	"\u001b[37m File \"tenacity\\asyncio\\__call__.py\", line 112, in __call__\u001b[0m\n",
	"\u001b[37m do = await self.iter(retry_state=retry_state)\u001b[0m\n",
	"\u001b[37m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\u001b[0m\n",
	"\u001b[37m File \"tenacity\\asyncio\\__iter__.py\", line 1, in iter\u001b[0m\n",
	"\u001b[37m # redacted local environment path\u001b[0m\n",
	"\u001b[37m\u001b[0m\n",

[Copilot]

There is an inline import of get_azure_openai_auth inside OpenAITarget.__init__. Since azure-identity is a core dependency, this can be imported at module scope to keep imports at the top of the file and avoid deferring import errors until runtime.

[Copilot]

Azure endpoint detection uses "azure" in endpoint_value.lower(), which can misclassify valid Azure endpoints that don't contain the literal substring (e.g., custom domains) and force API-key auth unnecessarily. Consider using urlparse(endpoint_value).netloc.endswith((".openai.azure.com", ".models.ai.azure.com", ".ai.azure.com")) (or reuse pyrit.auth.get_default_azure_scope logic) to decide when to fall back to Entra ID auth.

[romanlutz]

TODO @romanlutz to check correctness

[Copilot]

Pull request overview

Copilot reviewed 25 out of 29 changed files in this pull request and generated 8 comments.

[Copilot]

This notebook diff adds extensive stderr/stdout output (including content-filter / rate-limit logs and other run artifacts). For documentation notebooks, it’s usually better to clear outputs before committing so the repo doesn’t accumulate volatile execution logs and potentially sensitive information.

[Copilot]

PromptChatTarget.supports_multi_turn always returns True, which bypasses TargetCapabilities and makes per-instance overrides (via the capabilities setter/constructor) ineffective. This will also cause tests like test_constructor_override_supports_multi_turn (which expects an override to False) to fail. Consider removing this override entirely and relying on _DEFAULT_CAPABILITIES, or delegating to super().supports_multi_turn / self.capabilities.supports_multi_turn.

Suggested change

	bool: True for chat targets.
	"""
	return True
	bool: True for chat targets by default, unless overridden via capabilities.
	"""
	return self.capabilities.supports_multi_turn

[Copilot]

This adds a per-instance capability mechanism, but supports_multi_turn is still hard-coded to True here. That prevents users/tests from overriding capabilities for this target (e.g., for debugging or unusual deployments) and duplicates what _DEFAULT_CAPABILITIES already expresses. Consider removing this property override and relying on PromptTarget.supports_multi_turn (capabilities-based).

Suggested change

	@property
	def supports_multi_turn(self) -> bool:
	"""
	Playwright targets maintain state via browser page across turns.
	"""
	return True

[Copilot]

Same as other targets: this supports_multi_turn property is hard-coded to True, which bypasses TargetCapabilities overrides and duplicates _DEFAULT_CAPABILITIES. Consider delegating to self.capabilities.supports_multi_turn (or removing the override) so the new capability system behaves consistently across targets.

Suggested change

	return True
	return self.capabilities.supports_multi_turn

[Copilot]

Inline import of get_azure_openai_auth inside __init__ violates the project’s import-at-top convention and makes dependency analysis harder. Since pyrit.auth.azure_auth only references pyrit.prompt_target in docstrings, this likely isn’t needed to avoid a circular import—prefer importing get_azure_openai_auth at module scope (or add a comment explaining the circular dependency if it’s real).

[Copilot]

AzureContentFilterScorer.__init__ accepts token-provider callables that may return an Awaitable[str], but TokenProviderCredential.get_token() (used below) calls the provider synchronously and coerces the result via str(token), which will mis-handle async token providers. Consider either (a) tightening the accepted type to synchronous Callable[[], str] and rejecting coroutine functions, or (b) updating the credential wrapper/scorer initialization path to correctly await async providers.

[Copilot]

The validation now allows converted_value_data_type to be "error", but the raised message still says this method “only supports text.” Consider updating the exception message (and docstring) to reflect the actual accepted types (or explicitly skip error messages instead of sending them back to the model).