[Bug]: A conflict occurs when crawler_strategy and screenshot=True are set together

[OpenAI-Insights]

crawl4ai version

0.7.8

Expected Behavior

When using crawl4ai for web crawling, Stealth Mode and Undetected Browser Mode are typically enabled. The crawled content can be returned as Markdown or HTML without any issues. However, an error occurs when setting the screenshot parameter in config=CrawlerRunConfig(screenshot=True), as shown below:

Current Behavior

The core cause of this error is as follows: the self.logger object used in the code is None (a null value), but the code attempts to call its .info() method, which triggers the error: 'NoneType' object has no attribute 'info'.
Modify the code around line 1005 in the crawl4ai source code to add a non-null check for the logger object, and update the file site-packages/crawl4ai/async_crawler_strategy.py.

Lines 1004 to 1009 of the original code
if screenshot_data or pdf_data or mhtml_data:
self.logger.info(
message="Exporting media (PDF/MHTML/screenshot) took {duration:.2f}s",
tag="EXPORT",
params={"duration": time.perf_counter() - start_export_time},
)

After modification

if screenshot_data or pdf_data or mhtml_data:
if self.logger is not None: # Add non-null check
self.logger.info(
message="Exporting media (PDF/MHTML/screenshot) took {duration:.2f}s",
tag="EXPORT",
params={"duration": time.perf_counter() - start_export_time},
)

This change enables my code to run properly.

Is this reproducible?

Yes

Inputs Causing the Bug

Steps to Reproduce

Code snippets

async def main():

    browser_config = BrowserConfig(
        enable_stealth=True,  # Enable stealth mode
        headless=True,
        headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36 Edg/143.0.0.0"}
    )

    adapter = UndetectedAdapter()

    strategy = AsyncPlaywrightCrawlerStrategy(
        browser_config=browser_config,
        browser_adapter=adapter,
    )

    crawler_config = CrawlerRunConfig(
        wait_for_images=True,
        scan_full_page=True,
        scroll_delay=0.5,
        cache_mode=CacheMode.BYPASS,
        excluded_tags=["form", "header", "footer"],
        keep_data_attributes=False,
        screenshot=True
        
    )
    
    async with AsyncWebCrawler(
        crawler_strategy=strategy,
        config=browser_config,
    ) as crawler:
        result = await crawler.arun(
            url="",
            config=crawler_config,
        )

        output_path = "article.md"
        with open(output_path, "w", encoding="utf-8") as f:
            f.write(result.markdown)
        
        output_path = "article.html"
        with open(output_path, "w", encoding="utf-8") as f:
            f.write(result.cleaned_html)
        if result.success:
            print("✓ Successfully accessed bot detection test site")
            # Save screenshot to verify detection results
            if result.screenshot:
                with open("stealth_test2.png", "wb") as f:
                    f.write(base64.b64decode(result.screenshot))
                print("✓ Screenshot saved - check for green (passed) tests")

OS

Windows wsl

Python version

3.12

Browser

No response

Browser version

No response

Error logs & Screenshots (if applicable)

[INIT].... → Crawl4AI 0.7.8
[ERROR]... × https://mp.weixin.qq.com/s/M1vGqFZV5MWREkSyx2-ITw | Error: Unexpected error in _crawl_web at
line 1005 in _crawl_web
(../../../../../miniconda3/envs/rag/lib/python3.12/site-packages/crawl4ai/async_crawler_strategy.py):
Error: 'NoneType' object has no attribute 'info'

Code context:
1000 screenshot_data = await self.take_screenshot(
1001 page, screenshot_height_threshold=config.screenshot_height_threshold
1002 )
1003
1004 if screenshot_data or pdf_data or mhtml_data:
1005 → self.logger.info(
1006 message="Exporting media (PDF/MHTML/screenshot) took {duration:.2f}s",
1007 tag="EXPORT",
1008 params={"duration": time.perf_counter() - start_export_time},
1009 )
1010
Traceback (most recent call last):
File "/home/edy/data/project/HAICHENG_QA/code/src/html2images.py", line 80, in
asyncio.run(main())
File "/home/edy/miniconda3/envs/rag/lib/python3.12/asyncio/runners.py", line 195, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/home/edy/miniconda3/envs/rag/lib/python3.12/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/edy/miniconda3/envs/rag/lib/python3.12/asyncio/base_events.py", line 691, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/home/edy/data/project/HAICHENG_QA/code/src/html2images.py", line 59, in main
f.write(result.markdown)
TypeError: write() argument must be str, not None

[Ahmed-Tawfik94]

Root Cause Analysis

1. Direct Instantiation of AsyncPlaywrightCrawlerStrategy

The issue occurs when users directly instantiate AsyncPlaywrightCrawlerStrategy and pass it to AsyncWebCrawler:

# User's code from the issue
strategy = AsyncPlaywrightCrawlerStrategy(
    browser_config=browser_config,
    browser_adapter=adapter,
)

async with AsyncWebCrawler(
    crawler_strategy=strategy,  # Strategy passed without logger
    config=browser_config,
) as crawler:
    result = await crawler.arun(url="...", config=crawler_config)

2. Logger Initialization Flow

When examining the code flow:

In async_webcrawler.py (lines 137-151):

# Logger is created FIRST
self.logger = logger or AsyncLogger(
    log_file=os.path.join(base_directory, ".crawl4ai", "crawler.log"),
    verbose=self.browser_config.verbose,
    tag_width=10,
)

# Then strategy is initialized
self.crawler_strategy = crawler_strategy or AsyncPlaywrightCrawlerStrategy(
    browser_config=browser_config,
    logger=self.logger,  # Logger passed here
    **params,
)

However, when a user provides a pre-instantiated strategy:

# User creates strategy WITHOUT logger
strategy = AsyncPlaywrightCrawlerStrategy(
    browser_config=browser_config,
    browser_adapter=adapter,
)

# AsyncWebCrawler receives pre-instantiated strategy
async with AsyncWebCrawler(crawler_strategy=strategy) as crawler:
    # The strategy already has logger=None from initialization
    # AsyncWebCrawler doesn't update the strategy's logger

In async_crawler_strategy.py (lines 74-90):

def __init__(
    self, browser_config: BrowserConfig = None, logger: AsyncLogger = None,
    browser_adapter: BrowserAdapter = None, **kwargs
):
    self.browser_config = browser_config or BrowserConfig.from_kwargs(kwargs)
    self.logger = logger  # If not provided, this is None
    self.adapter = browser_adapter or PlaywrightAdapter()

3. Inconsistent Null Checks

The codebase has inconsistent patterns for logger usage:

Places with null checks (8 occurrences):

if self.logger:
    self.logger.warning(...)

Places WITHOUT null checks (47 occurrences):

self.logger.info(...)  # Crashes if logger is None

4. Why Screenshot Triggers the Issue

The error specifically occurs at line 1021 when screenshot/PDF/MHTML export completes:

# Lines 1020-1025
if screenshot_data or pdf_data or mhtml_data:
    self.logger.info(  # ← CRASH HERE if logger is None
        message="Exporting media (PDF/MHTML/screenshot) took {duration:.2f}s",
        tag="EXPORT",
        params={"duration": time.perf_counter() - start_export_time},
    )

This is one of the many places where logger is used without a null check.

Impact Analysis

Affected Scenarios

1. ✅ Works: Using AsyncWebCrawler() without custom strategy
2. ✅ Works: Using AsyncWebCrawler(logger=custom_logger) without custom strategy
3. ❌ Fails: Using AsyncWebCrawler(crawler_strategy=custom_strategy) where strategy has no logger
4. ❌ Fails: Direct instantiation of AsyncPlaywrightCrawlerStrategy without logger

Severity

• High: Breaks core functionality (screenshot, PDF, MHTML export)
• User Impact: Users following the documented pattern for custom strategies encounter crashes
• Workaround: Users must manually pass logger when creating strategy

[blueemp]

Analysis and Solution

Root Cause

The AsyncPlaywrightCrawlerStrategy.__init__ method accepts a logger parameter that defaults to None. When users create the strategy without explicitly passing a logger, self.logger remains None, causing the error when self.logger.info() is called at line 1005.

Recommended Fix

Initialize a default logger in the __init__ method if none is provided. Modify async_crawler_strategy.py around line 82:

def __init__(
    self, browser_config: BrowserConfig = None, logger: AsyncLogger = None, browser_adapter: BrowserAdapter = None, **kwargs
):
    self.browser_config = browser_config or BrowserConfig.from_kwargs(kwargs)
    # Initialize with default logger if none provided
    self.logger = logger if logger is not None else AsyncLogger(verbose=False)
    
    # Initialize browser adapter
    self.adapter = browser_adapter or PlaywrightAdapter()
    # ... rest of initialization

This ensures the strategy always has a valid logger and prevents similar issues in the future. The default logger can be configured to be silent or use standard Python logging.

Alternative Fix (Your Suggestion)

Add null checks before every self.logger call, but this is less maintainable and doesn't prevent future issues.