[Bug]: CosineStrategy not generating results

[lmseidler]

crawl4ai version

0.7.4

Expected Behavior

extractor = CosineStrategy("some filter")
clusters = extractor.extract(url, html) <-- should return a list containing the clustering results

Current Behavior

clusters = extractor.extract(url, html) <-- always returns an empty list

Only works if e.g extractor.DEL = "\n"

The reason for the bug is apparently that CosineStrategy.DEL doesn't point to an appropriate character. Therefore, the html.split in the beginning of extract fails to produce more than one item. In filter_document_embeddings an empty list gets returned because at_least_k = 1 // 2 = 0. extract then proceeds to return an empty list as output.

Is this reproducible?

Yes

Inputs Causing the Bug

Steps to Reproduce

Take any URL and HTML content and use CosineStrategy.extract. Will always return an empty list.

Code snippets

OS

linux

Python version

3.11

Browser

No response

Browser version

No response

Error logs & Screenshots (if applicable)

No response

[Ahmed-Tawfik94]

Summary

Aspect	Details
Issue	CosineStrategy.extract() always returns an empty list
Scope	All usages of CosineStrategy with default settings
Root Cause	Default delimiter `<
Fix	Set appropriate delimiter or modify initialization logic
Risk	Low - fix doesn't affect other strategies, isolated to CosineStrategy

What's Happening

The CosineStrategy.extract() method consistently returns an empty list when processing HTML or markdown content, preventing any content extraction or clustering from occurring.

What's Affected

• All direct uses of CosineStrategy with default initialization
• Any workflows depending on CosineStrategy for content clustering and extraction
• Users following documentation examples without custom delimiter configuration

What's Not Affected

• Other extraction strategies (NoExtractionStrategy, LLMExtractionStrategy, etc.)
• CosineStrategy instances where users explicitly set extractor.DEL = "\n" or similar
• Other crawler functionality independent of CosineStrategy

---

Technical Background

Current Implementation

The CosineStrategy class is designed to extract meaningful content blocks through hierarchical clustering of text chunks. The process follows this flow:

HTML Content → Split by DEL → Filter Embeddings → Cluster → Extract Results

Relevant code from extraction_strategy.py:

class ExtractionStrategy(ABC):
    def __init__(self, input_format: str = "markdown", **kwargs):
        self.input_format = input_format
        self.DEL = "<|DEL|>"  # Default delimiter
        self.name = self.__class__.__name__
        self.verbose = kwargs.get("verbose", False)

CosineStrategy.extract() from extraction_strategy.py:

def extract(self, url: str, html: str, *q, **kwargs) -> List[Dict[str, Any]]:
    """Extract clusters from HTML content using hierarchical clustering."""
    t = time.time()
    text_chunks = html.split(self.DEL)  # Split by delimiter
    
    # Pre-filter documents using embeddings and semantic_filter
    text_chunks = self.filter_documents_embeddings(
        text_chunks, self.semantic_filter
    )
    
    if not text_chunks:
        return []  # Returns empty if filtering yields no chunks
    
    # ... clustering logic follows ...

filter_documents_embeddings() from extraction_strategy.py:

def filter_documents_embeddings(
    self, documents: List[str], semantic_filter: str, at_least_k: int = 20
) -> List[str]:
    """Filter and sort documents based on cosine similarity."""
    
    if not semantic_filter:
        return documents
    
    if len(documents) < at_least_k:
        at_least_k = len(documents) // 2  # Critical line: 1 // 2 = 0
    
    # ... embedding calculations ...
    
    return filtered_docs[:at_least_k]  # Returns [:0] when at_least_k=0

Design Intent

The DEL delimiter was designed to split pre-chunked content that had been joined with a known separator. However, the assumption that content would contain this delimiter doesn't hold for typical HTML/markdown inputs.

---

What Changed

No explicit code changes caused this issue. The behavior has been present since the CosineStrategy implementation, but the constraint became apparent when users began applying it to raw HTML content rather than pre-processed, delimiter-separated text.

Flow diagram showing where the issue occurs:

Input HTML
    ↓
html.split("<|DEL|>")  ← HTML doesn't contain delimiter
    ↓
[single chunk]  ← Only 1 element in list
    ↓
filter_documents_embeddings()
    ↓
at_least_k = 1 // 2 = 0  ← Integer division results in 0
    ↓
filtered_docs[:0]  ← Slicing to index 0 returns empty list
    ↓
extract() checks: if not text_chunks
    ↓
return []  ← Empty result

---

Context

Historical Design Decisions

The CosineStrategy implementation initializes from a base class where DEL = "<|DEL|>" serves as a general delimiter for joining content sections. This pattern works well when:

1. Content is pre-processed and joined with the delimiter
2. Users explicitly set the delimiter to match their content structure
3. Multiple sections are already separated by this specific string

Why This Affects Normal Usage

When users apply CosineStrategy directly to HTML or markdown content (as shown in documentation examples), this delimiter doesn't naturally appear in the text. The split operation then returns a single-element list containing the entire HTML content.

Related Code Patterns

Other strategies in the same file handle content differently:

• NoExtractionStrategy: Returns content as-is without splitting
• LLMExtractionStrategy: Processes content in sections differently

The CosineStrategy uniquely depends on the delimiter for initial chunking, making it vulnerable to this edge case.

---