[Bug]: Docker API + Deep crawling + Stream not working

[Qbason]

crawl4ai version

0.8

Expected Behavior

Docker API + Deep crawling + Stream should work

Current Behavior

Docker API + Deep crawling + Stream doesn't work:
"error_message": "'async_generator' object has no attribute 'status_code'",

Is this reproducible?

Yes

Inputs Causing the Bug

Steps to Reproduce

[docker](https://github.com/unclecode/crawl4ai/tree/main/docs/examples/docker)
/demo_docker_api.py
method -> demo_deep_streaming

Code snippets

OS

Linux, Docker

Python version

Browser

No response

Browser version

No response

Error logs & Screenshots (if applicable)

No response

[SohamKukreti]

Hey, thanks for raising this issue, I have managed to reproduce it and root cause it as well:

Root Cause Analysis

Summary

Deep crawling + streaming via the Docker /crawl/stream API is broken because the dispatcher assumes AsyncWebCrawler.arun() returns a single CrawlResult, but in deep-crawl streaming mode arun() actually returns an async_generator. When the dispatcher then tries to read result.status_code, it's operating on an async_generator, causing the error. That exception is caught and surfaced back in the streamed JSON as the error_message seen by users.

Technical Background

• Users call the Docker API with:

  • crawler_config.deep_crawl_strategy = BFSDeepCrawlStrategy(...)
  • crawler_config.stream = True
  • Endpoint: POST /crawl/stream (via demo_deep_streaming in docs/examples/docker/demo_docker_api.py)

• The request path in the Docker image is:

  1. POST /crawl/stream → server.stream_process(...) which calls handle_stream_crawl_request(...) and returns StreamingResponse(stream_results(crawler, gen), ...)
  2. handle_stream_crawl_request(...) loads config, sets stream=True, gets a crawler from the pool, and calls:

  results_gen = await crawler.arun_many(
      urls=urls,
      config=crawler_config,
      dispatcher=dispatcher
  )

  3. AsyncWebCrawler.arun_many(..., stream=True) returns an async generator that consumes from dispatcher.run_urls_stream(crawler=self, urls=urls, config=config)
  4. MemoryAdaptiveDispatcher.run_urls_stream(...) internally calls crawl_url(...) for each URL, which does:

  result = await self.crawler.arun(url, config=selected_config, session_id=task_id)
  ...
  if self.rate_limiter and result.status_code:
      ...

• Deep crawling is implemented by decorating AsyncWebCrawler.arun:

  self._deep_handler = DeepCrawlDecorator(self)
  self.arun = self._deep_handler(self.arun)

• When config.deep_crawl_strategy is set and config.stream is True, the DeepCrawlDecorator in deep_crawling/base_strategy.py does:

  result_obj = await config.deep_crawl_strategy.arun(
      crawler=self.crawler,
      start_url=url,
      config=config,
  )
  if config.stream:
      async def result_wrapper():
          try:
              async for result in result_obj:
                  yield result
          finally:
              self.deep_crawl_active.reset(token)
      return result_wrapper()  # returns an async generator

• For deep crawl + stream:

  • config.deep_crawl_strategy is set
  • config.stream is True
  • DeepCrawlStrategy.arun(...) returns an async generator (from its _arun_stream implementation)
  • After await, result_obj is that async generator
  • So AsyncWebCrawler.arun(...) returns result_wrapper() — an async generator

• The dispatcher, however, assumes arun returns a single CrawlResult and immediately does:

  result = await self.crawler.arun(url, config=selected_config, session_id=task_id)
  ...
  if self.rate_limiter and result.status_code:
      ...

  Here result is that async generator. Calling result.status_code raises:

  AttributeError: 'async_generator' object has no attribute 'status_code'

• That exception is caught in the dispatcher and turned into a failed CrawlResult:

  except Exception as e:
      error_message = str(e)
      result = CrawlResult(
          url=url, html="", metadata={},
          success=False, error_message=str(e),
      )

• This CrawlResult flows through arun_many → run_urls_stream → stream_results(...), which serializes it via model_dump() and emits it as NDJSON. That's why the user sees JSON like:

  {
    "url": "...",
    "success": false,
    "error_message": "'async_generator' object has no attribute 'status_code'",
    "dispatch_result": { "error_message": "...", ... },
    ...
  }

---

Root Cause (Direct)

1. API contract mismatch

   • AsyncWebCrawler.arun has an overloaded return type:
     • Normal mode: returns a single CrawlResult
     • Deep-crawl + streaming: returns an async_generator[CrawlResult]
   • The dispatcher (MemoryAdaptiveDispatcher.crawl_url) was written assuming the single-result contract and unconditionally treats the return value as a CrawlResult object, accessing attributes like .status_code.

2. Interaction specific to Docker deep streaming

   • Docker /crawl/stream always uses stream=True and routes through handle_stream_crawl_request, which always uses MemoryAdaptiveDispatcher.
   • When a deep-crawl strategy is also configured, the decorator changes the semantics of arun from "single result" to "stream of results", but the dispatcher code path was never updated to handle this.

3. Error surfacing

   • The dispatcher catches all exceptions and converts them into CrawlResult(success=False, error_message=str(e)), which is then streamed. So users get a valid NDJSON stream with failed results rather than an HTTP 500, which can make the bug less obvious.