[Bug]: Browser hangs indefinitely on WSL when system proxy is required

[nihaoWX]

crawl4ai version

0.8.6

Expected Behavior

When proxy or proxy_config is set in BrowserConfig, crawl4ai should
successfully launch Chromium with the proxy and complete the crawl.

Current Behavior

On WSL2 where a proxy is required for internet access, AsyncWebCrawler hangs
indefinitely after printing "[INIT].... → Crawl4AI 0.8.6" and never completes.
Setting proxy via BrowserConfig(proxy=...), BrowserConfig(proxy_config=...),
or extra_args=["--proxy-server=..."] all result in the same hang.

Is this reproducible?

Yes

Inputs Causing the Bug

Any URL requiring internet access (e.g. https://example.com) when running on 
WSL2 where Chrome cannot connect to the internet directly.

Steps to Reproduce

1. Install crawl4ai on WSL2:
   pip install crawl4ai
   crawl4ai-setup

2. Verify that Python requests work fine with proxy (direct Chrome cannot reach internet):
   HOST_IP=$(ip route | awk '/default/ {print $3}')
   http_proxy=http://$HOST_IP:7892 python -c "import requests; print(requests.get('https://www.google.com', timeout=15).status_code)"
   # Output: 200

3. Verify that plain Playwright works fine with proxy:
   python -c "
   from playwright.sync_api import sync_playwright
   p = sync_playwright().start()
   b = p.chromium.launch()
   print('OK')
   b.close()
   p.stop()
   "
   # Output: OK

4. Run a basic crawl4ai crawl (no proxy):
   python -c "
   import asyncio
   from crawl4ai import AsyncWebCrawler
   async def test():
       async with AsyncWebCrawler() as crawler:
           result = await crawler.arun('https://example.com')
           print(result.success)
   asyncio.run(test())
   "
   # Hangs forever at: [INIT].... → Crawl4AI 0.8.6

5. Try with proxy set in BrowserConfig:
   python -c "
   import asyncio
   from crawl4ai import AsyncWebCrawler, BrowserConfig
   async def test():
       config = BrowserConfig(proxy='http://172.29.240.1:7892')
       async with AsyncWebCrawler(config=config) as crawler:
           result = await crawler.arun('https://example.com')
           print(result.success)
   asyncio.run(test())
   "
   # Also hangs forever at: [INIT].... → Crawl4AI 0.8.6

6. Confirmed root cause: crawl4ai uses subprocess.Popen to launch Chrome 
   without passing proxy env vars, so Chrome cannot reach the internet on WSL2.
   Plain Playwright with proxy={"server": ...} works correctly.

Code snippets

# Hangs forever
import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig

async def main():
    config = BrowserConfig(proxy="http://172.29.240.1:7892")
    async with AsyncWebCrawler(config=config) as crawler:
        result = await crawler.arun("https://example.com")

asyncio.run(main())

# Workaround: launch browser manually via Playwright and connect via CDP
import subprocess
from playwright.async_api import async_playwright
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig

HOST_IP = subprocess.check_output(
    "ip route | awk '/default/ {print $3}'", shell=True).decode().strip()

async def main():
    async with async_playwright() as p:
        browser = await p.chromium.launch(
            headless=True,
            args=["--no-sandbox", "--disable-dev-shm-usage",
                  "--remote-debugging-port=9222"],
            proxy={"server": f"http://{HOST_IP}:7892"}
        )
        config = BrowserConfig(cdp_url="http://localhost:9222")
        async with AsyncWebCrawler(config=config) as crawler:
            result = await crawler.arun("https://example.com",
                config=CrawlerRunConfig(page_timeout=15000))
            print(result.success)
        await browser.close()

asyncio.run(main())

OS

OS: Windows 11 + WSL2 (Ubuntu 24.04)

Python version

3.12

Browser

Chromium (crawl4ai managed)

Browser version

Google Chrome for Testing 145.0.7632.6

Error logs & Screenshots (if applicable)

No response

[nihaoWX]

Workaround (working solution)

Launch Chromium manually via Playwright with proxy, then connect crawl4ai via CDP URL:

import asyncio
import subprocess
from playwright.async_api import async_playwright
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig

HOST_IP = subprocess.check_output(
    "ip route | awk '/default/ {print $3}'", shell=True).decode().strip()
PROXY = f"http://{HOST_IP}:7892"

async def main():
    async with async_playwright() as p:
        # Launch Chrome via Playwright (proxy is correctly passed)
        browser = await p.chromium.launch(
            headless=True,
            args=["--no-sandbox", "--disable-dev-shm-usage",
                  "--remote-debugging-port=9222"],
            proxy={"server": PROXY}
        )
        # Pass CDP URL to crawl4ai so it skips its own browser launch
        config = BrowserConfig(cdp_url="http://localhost:9222")
        async with AsyncWebCrawler(config=config) as crawler:
            result = await crawler.arun(
                "https://example.com",
                config=CrawlerRunConfig(page_timeout=15000)
            )
            print("success:", result.success)
            print(result.markdown[:200])
        await browser.close()

asyncio.run(main())

Output:

proxy: http://172.29.240.1:7892
browser launched, cdp: http://localhost:9222
[INIT].... → Crawl4AI 0.8.6
[FETCH]... ↓ https://example.com  | ✓ | ⏱: 0.50s
[SCRAPE].. ◆ https://example.com  | ✓ | ⏱: 0.00s
[COMPLETE] ● https://example.com  | ✓ | ⏱: 0.50s
success: True
# Example Domain
This domain is for use in documentation examples without needing permission.

Note: The RuntimeError: Event loop is closed at the end is a harmless asyncio
cleanup warning and does not affect functionality.

[ntohidi]

@nihaoWX Thanks for the detailed report and for finding the workaround!

This is a WSL2-specific network issue rather than a code bug. The proxy is being passed to Chrome via the --proxy-server CLI flag, but WSL2's network bridging can cause issues with how the subprocess resolves the proxy host IP during browser startup.

Besides the CDP workaround you found, there's a simpler option — bypass the subprocess launch and use Playwright's native browser launch instead:

config = BrowserConfig(
    proxy_config=ProxyConfig("http://172.29.240.1:7892"),
    use_managed_browser=False,
    browser_mode="dedicated",
)
async with AsyncWebCrawler(config=config) as crawler:
    result = await crawler.arun("https://example.com")

This forces Playwright to handle the proxy at the protocol level (which is why your CDP workaround also works), instead of passing it as a CLI flag to a subprocess.