Crawling Time Increasing Over Time (| ✓ | ⏱: 127.83s Even for Simple Pages)

[dvirginz]

Hi team,

I’ve noticed that recently the crawling process with Crawl4AI takes quite a bit longer than it used to — even for very simple pages. For example, a crawl now reports | ✓ | ⏱: 127.83s, which feels unusually slow.

I completely understand that this likely has nothing to do directly with Crawl4AI itself, but I wanted to ask if you have any idea what could cause this kind of slowdown? Maybe something like caching issues, resource piling up over time, or anything else I should check?

It wasn’t like this when I started using it, so I’m trying to figure out what could have changed.

Thanks in advance!

[hafezparast]

There have been several performance and memory-related fixes in recent releases that may help. Here's what I'd check:

1. Upgrade to the latest version (v0.8.6)
Several fixes were made since this was posted:

• Browser context memory leak fix (context LRU eviction)
• Browser recycling deadlock fix under concurrent load
• scan_full_page hanging fix on dynamic/infinite-scroll pages
• Memory-saving mode and CDP leak fixes

2. Check your CrawlerRunConfig settings

config = CrawlerRunConfig(
    scan_full_page=False,       # True causes full-page scrolling — can be very slow
    wait_for_images=False,      # Waiting for all images adds time
    page_timeout=60000,         # Default is 60s — lower if your pages are simple
    wait_until="domcontentloaded",  # Faster than "networkidle"
)

3. Browser context accumulation
If you're crawling many URLs in a loop with the same AsyncWebCrawler instance, browser contexts can pile up. Use the async context manager and consider setting session_id to reuse contexts, or create a fresh crawler periodically.

4. Resource filtering
Block unnecessary resources to speed things up:

browser_config = BrowserConfig(
    extra_args=["--disable-images", "--disable-gpu"]
)

If the slowdown correlates with how many pages you've crawled (gets worse over time in the same session), it's almost certainly the context/memory accumulation issue — upgrading should fix it.