Skip to content

fix(docker): make deep-crawl streaming mirror Python library behaviour#1798

Open
SohamKukreti wants to merge 1 commit intodevelopfrom
fix/deep-crawl-stream-docker
Open

fix(docker): make deep-crawl streaming mirror Python library behaviour#1798
SohamKukreti wants to merge 1 commit intodevelopfrom
fix/deep-crawl-stream-docker

Conversation

@SohamKukreti
Copy link
Collaborator

@SohamKukreti SohamKukreti commented Mar 3, 2026

Summary

Changes made in the PR:

  • Route deep-crawl stream=True requests with a single URL through AsyncWebCrawler.arun so each discovered page is streamed as its own CrawlResult
  • Preserve existing arun_many + MemoryAdaptiveDispatcher behavior for non–deep-crawl or multi-URL streaming.
  • Add Docker REST tests for deep-crawl streaming success (single URL) and helpful error on multi-URL usage

Fixes: (#1779)

List of files changed and why

deploy/docker/api.py: Mirror python library deep crawl stream behaviour in handle_stream_crawl_request()
tests/docker/test_rest_api_deep_crawl.py: Add test for deep crawl stream behaviour

How Has This Been Tested?

Built and ran the docker container, ran custom example script and created new tests for testing functionality.

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have added/updated unit tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

#1779)

- Route deep-crawl stream=True requests with a single URL through AsyncWebCrawler.arun so each discovered page is streamed as its own CrawlResult
- Preserve existing arun_many + MemoryAdaptiveDispatcher behavior for non–deep-crawl or multi-URL streaming.
- Add Docker REST tests for deep-crawl streaming success (single URL) and helpful error on multi-URL usage
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant