feat: add tests for cohere by shellmayr · Pull Request #63 · getsentry/testing-ai-sdk-integrations

shellmayr · 2026-02-27T11:31:11Z

Add tests for cohere
We need two templates, because there are two versions of the API that both need to be supported

Closes TET-2025

github-actions · 2026-02-27T11:40:13Z

🟡 AI SDK Integration Test Results

Status: 1 test fixed, 236 still failing

Summary

Metric	main	PR	Change
Total Tests	448	464	+16
Passed	227	228	+1 ✅
Failed	221	236	+15 ⚠️

✅ Fixed

These tests were failing on main but are now passing:

cloudflare/google-genai :: Vision LLM Test (blocking)

🆕 New Tests

Failing (16):

❌ python/cohere :: Basic LLM Test (sync, streaming)

Error: 3 check(s) failed:

3 check(s) failed:
Should have exactly 1 AI span(s) but found 0
Should have at least one chat/completion span
Should have at least one chat or agent span

❌ python/cohere :: Basic LLM Test (sync, blocking)

Error: 3 check(s) failed:

3 check(s) failed:
Should have exactly 1 AI span(s) but found 0
Should have at least one chat/completion span
Should have at least one chat or agent span

❌ python/cohere :: Multi-Turn LLM Test (sync, streaming)

Error: 3 check(s) failed:

3 check(s) failed:
Should have exactly 3 AI span(s) but found 0
Should have at least one chat/completion span
Should have at least one chat or agent span

❌ python/cohere :: Multi-Turn LLM Test (sync, blocking)

Error: Test execution failed: Command failed: /home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-5.13.0-sentry-latest/.venv/bin/python /home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-5.13.0-sentry-latest/test-multi-turn-llm-test-sync-blocking.py

Test execution failed: Command failed: /home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-5.13.0-sentry-latest/.venv/bin/python /home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-5.13.0-sentry-latest/test-multi-turn-llm-test-sync-blocking.py
Traceback (most recent call last):
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-5.13.0-sentry-latest/test-multi-turn-llm-test-sync-blocking.py", line 88, in <module>
    main()
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-5.13.0-sentry-latest/test-multi-turn-llm-test-sync-blocking.py", line 83, in main
    response = client.chat(**kwargs)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-5.13.0-sentry-latest/.venv/lib/python3.12/site-packages/cohere/client.py", line 103, in _wrapped
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-5.13.0-sentry-latest/.venv/lib/python3.12/site-packages/cohere/client.py", line 35, in _wrapped
    return method(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-5.13.0-sentry-latest/.venv/lib/python3.12/site-packages/sentry_sdk/integrations/cohere.py", line 159, in new_chat
    reraise(*exc_info)
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-5.13.0-sentry-latest/.venv/lib/python3.12/site-packages/sentry_sdk/utils.py", line 1785, in reraise
    raise value
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-5.13.0-sentry-latest/.venv/lib/python3.12/site-packages/sentry_sdk/integrations/cohere.py", line 153, in new_chat
    res = f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-5.13.0-sentry-latest/.venv/lib/python3.12/site-packages/cohere/base_client.py", line 1111, in chat
    raise TooManyRequestsError(
cohere.errors.too_many_requests_error.TooManyRequestsError: status_code: 429, body: data=None id='ccf222f8-4b53-4f95-8d19-288df85a1ab2' message="You are using a Trial key, which is limited to 20 API calls / minute. You can continue to use the Trial key for free or upgrade to a Production key with higher rate limits at 'https://dashboard.cohere.com/api-keys'. Contact us on 'https://discord.gg/XW44jPfYJu' or email us at support@cohere.com with any questions"

Traceback (most recent call last):
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-5.13.0-sentry-latest/test-multi-turn-llm-test-sync-blocking.py", line 88, in <module>
    main()
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-5.13.0-sentry-latest/test-multi-turn-llm-test-sync-blocking.py", line 83, in main
    response = client.chat(**kwargs)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-5.13.0-sentry-latest/.venv/lib/python3.12/site-packages/cohere/client.py", line 103, in _wrapped
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-5.13.0-sentry-latest/.venv/lib/python3.12/site-packages/cohere/client.py", line 35, in _wrapped
    return method(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-5.13.0-sentry-latest/.venv/lib/python3.12/site-packages/sentry_sdk/integrations/cohere.py", line 159, in new_chat
    reraise(*exc_info)
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-5.13.0-sentry-latest/.venv/lib/python3.12/site-packages/sentry_sdk/utils.py", line 1785, in reraise
    raise value
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-5.13.0-sentry-latest/.venv/lib/python3.12/site-packages/sentry_sdk/integrations/cohere.py", line 153, in new_chat
    res = f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-5.13.0-sentry-latest/.venv/lib/python3.12/site-packages/cohere/base_client.py", line 1111, in chat
    raise TooManyRequestsError(
cohere.errors.too_many_requests_error.TooManyRequestsError: status_code: 429, body: data=None id='ccf222f8-4b53-4f95-8d19-288df85a1ab2' message="You are using a Trial key, which is limited to 20 API calls / minute. You can continue to use the Trial key for free or upgrade to a Production key with higher rate limits at 'https://dashboard.cohere.com/api-keys'. Contact us on 'https://discord.gg/XW44jPfYJu' or email us at support@cohere.com with any questions"

❌ python/cohere :: Basic Error LLM Test (sync, streaming)

Error: 1 check(s) failed:

1 check(s) failed:
Should have at least 1 AI span(s) but found 0

❌ python/cohere :: Basic Error LLM Test (sync, blocking)

Error: 1 check(s) failed:

1 check(s) failed:
Should have at least 1 AI span(s) but found 0

❌ python/cohere :: Conversation ID LLM Test (sync, streaming)

Error: 4 check(s) failed:

4 check(s) failed:
Should have exactly 4 AI span(s) but found 0
Should have at least one chat/completion span
Should have at least one AI span
Should have at least one chat or agent span

❌ python/cohere :: Conversation ID LLM Test (sync, blocking)

Error: Test execution failed: Command failed: /home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-5.13.0-sentry-latest/.venv/bin/python /home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-5.13.0-sentry-latest/test-conversation-id-llm-test-sync-blocking.py

Test execution failed: Command failed: /home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-5.13.0-sentry-latest/.venv/bin/python /home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-5.13.0-sentry-latest/test-conversation-id-llm-test-sync-blocking.py
Traceback (most recent call last):
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-5.13.0-sentry-latest/test-conversation-id-llm-test-sync-blocking.py", line 105, in <module>
    main()
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-5.13.0-sentry-latest/test-conversation-id-llm-test-sync-blocking.py", line 100, in main
    response = client.chat(**kwargs)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-5.13.0-sentry-latest/.venv/lib/python3.12/site-packages/cohere/client.py", line 103, in _wrapped
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-5.13.0-sentry-latest/.venv/lib/python3.12/site-packages/cohere/client.py", line 35, in _wrapped
    return method(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-5.13.0-sentry-latest/.venv/lib/python3.12/site-packages/sentry_sdk/integrations/cohere.py", line 159, in new_chat
    reraise(*exc_info)
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-5.13.0-sentry-latest/.venv/lib/python3.12/site-packages/sentry_sdk/utils.py", line 1785, in reraise
    raise value
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-5.13.0-sentry-latest/.venv/lib/python3.12/site-packages/sentry_sdk/integrations/cohere.py", line 153, in new_chat
    res = f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-5.13.0-sentry-latest/.venv/lib/python3.12/site-packages/cohere/base_client.py", line 1111, in chat
    raise TooManyRequestsError(
cohere.errors.too_many_requests_error.TooManyRequestsError: status_code: 429, body: data=None id='fc42724d-46f8-460a-b83b-338ea7cc2575' message="You are using a Trial key, which is limited to 20 API calls / minute. You can continue to use the Trial key for free or upgrade to a Production key with higher rate limits at 'https://dashboard.cohere.com/api-keys'. Contact us on 'https://discord.gg/XW44jPfYJu' or email us at support@cohere.com with any questions"

Traceback (most recent call last):
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-5.13.0-sentry-latest/test-conversation-id-llm-test-sync-blocking.py", line 105, in <module>
    main()
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-5.13.0-sentry-latest/test-conversation-id-llm-test-sync-blocking.py", line 100, in main
    response = client.chat(**kwargs)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-5.13.0-sentry-latest/.venv/lib/python3.12/site-packages/cohere/client.py", line 103, in _wrapped
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-5.13.0-sentry-latest/.venv/lib/python3.12/site-packages/cohere/client.py", line 35, in _wrapped
    return method(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-5.13.0-sentry-latest/.venv/lib/python3.12/site-packages/sentry_sdk/integrations/cohere.py", line 159, in new_chat
    reraise(*exc_info)
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-5.13.0-sentry-latest/.venv/lib/python3.12/site-packages/sentry_sdk/utils.py", line 1785, in reraise
    raise value
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-5.13.0-sentry-latest/.venv/lib/python3.12/site-packages/sentry_sdk/integrations/cohere.py", line 153, in new_chat
    res = f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-5.13.0-sentry-latest/.venv/lib/python3.12/site-packages/cohere/base_client.py", line 1111, in chat
    raise TooManyRequestsError(
cohere.errors.too_many_requests_error.TooManyRequestsError: status_code: 429, body: data=None id='fc42724d-46f8-460a-b83b-338ea7cc2575' message="You are using a Trial key, which is limited to 20 API calls / minute. You can continue to use the Trial key for free or upgrade to a Production key with higher rate limits at 'https://dashboard.cohere.com/api-keys'. Contact us on 'https://discord.gg/XW44jPfYJu' or email us at support@cohere.com with any questions"

❌ python/cohere-v2 :: Basic LLM Test (sync, streaming)

Error: 3 check(s) failed:

3 check(s) failed:
Should have exactly 1 AI span(s) but found 0
Should have at least one chat/completion span
Should have at least one chat or agent span

❌ python/cohere-v2 :: Basic LLM Test (sync, blocking)

Error: 3 check(s) failed:

3 check(s) failed:
Should have exactly 1 AI span(s) but found 0
Should have at least one chat/completion span
Should have at least one chat or agent span

❌ python/cohere-v2 :: Multi-Turn LLM Test (sync, streaming)

Error: Test execution failed: Command failed: /home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/.venv/bin/python /home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/test-multi-turn-llm-test-sync-streaming.py

Test execution failed: Command failed: /home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/.venv/bin/python /home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/test-multi-turn-llm-test-sync-streaming.py
Traceback (most recent call last):
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/test-multi-turn-llm-test-sync-streaming.py", line 82, in <module>
    main()
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/test-multi-turn-llm-test-sync-streaming.py", line 74, in main
    for event in stream:
                 ^^^^^^
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/.venv/lib/python3.12/site-packages/cohere/v2/client.py", line 333, in chat_stream
    raise TooManyRequestsError(
cohere.errors.too_many_requests_error.TooManyRequestsError: status_code: 429, body: data=None id='bebb7136-2b27-4ed7-b5d8-070a1d6d097d' message="You are using a Trial key, which is limited to 20 API calls / minute. You can continue to use the Trial key for free or upgrade to a Production key with higher rate limits at 'https://dashboard.cohere.com/api-keys'. Contact us on 'https://discord.gg/XW44jPfYJu' or email us at support@cohere.com with any questions"

Traceback (most recent call last):
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/test-multi-turn-llm-test-sync-streaming.py", line 82, in <module>
    main()
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/test-multi-turn-llm-test-sync-streaming.py", line 74, in main
    for event in stream:
                 ^^^^^^
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/.venv/lib/python3.12/site-packages/cohere/v2/client.py", line 333, in chat_stream
    raise TooManyRequestsError(
cohere.errors.too_many_requests_error.TooManyRequestsError: status_code: 429, body: data=None id='bebb7136-2b27-4ed7-b5d8-070a1d6d097d' message="You are using a Trial key, which is limited to 20 API calls / minute. You can continue to use the Trial key for free or upgrade to a Production key with higher rate limits at 'https://dashboard.cohere.com/api-keys'. Contact us on 'https://discord.gg/XW44jPfYJu' or email us at support@cohere.com with any questions"

❌ python/cohere-v2 :: Multi-Turn LLM Test (sync, blocking)

Error: Test execution failed: Command failed: /home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/.venv/bin/python /home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/test-multi-turn-llm-test-sync-blocking.py

Test execution failed: Command failed: /home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/.venv/bin/python /home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/test-multi-turn-llm-test-sync-blocking.py
Traceback (most recent call last):
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/test-multi-turn-llm-test-sync-blocking.py", line 70, in <module>
    main()
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/test-multi-turn-llm-test-sync-blocking.py", line 61, in main
    response = client.chat(
               ^^^^^^^^^^^^
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/.venv/lib/python3.12/site-packages/cohere/client.py", line 103, in _wrapped
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/.venv/lib/python3.12/site-packages/cohere/client.py", line 35, in _wrapped
    return method(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/.venv/lib/python3.12/site-packages/cohere/v2/client.py", line 636, in chat
    raise TooManyRequestsError(
cohere.errors.too_many_requests_error.TooManyRequestsError: status_code: 429, body: data=None id='0799700e-e1a6-4992-a3dd-5ee59153c085' message="You are using a Trial key, which is limited to 20 API calls / minute. You can continue to use the Trial key for free or upgrade to a Production key with higher rate limits at 'https://dashboard.cohere.com/api-keys'. Contact us on 'https://discord.gg/XW44jPfYJu' or email us at support@cohere.com with any questions"

Traceback (most recent call last):
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/test-multi-turn-llm-test-sync-blocking.py", line 70, in <module>
    main()
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/test-multi-turn-llm-test-sync-blocking.py", line 61, in main
    response = client.chat(
               ^^^^^^^^^^^^
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/.venv/lib/python3.12/site-packages/cohere/client.py", line 103, in _wrapped
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/.venv/lib/python3.12/site-packages/cohere/client.py", line 35, in _wrapped
    return method(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/.venv/lib/python3.12/site-packages/cohere/v2/client.py", line 636, in chat
    raise TooManyRequestsError(
cohere.errors.too_many_requests_error.TooManyRequestsError: status_code: 429, body: data=None id='0799700e-e1a6-4992-a3dd-5ee59153c085' message="You are using a Trial key, which is limited to 20 API calls / minute. You can continue to use the Trial key for free or upgrade to a Production key with higher rate limits at 'https://dashboard.cohere.com/api-keys'. Contact us on 'https://discord.gg/XW44jPfYJu' or email us at support@cohere.com with any questions"

❌ python/cohere-v2 :: Basic Error LLM Test (sync, streaming)

Error: 1 check(s) failed:

1 check(s) failed:
Should have at least 1 AI span(s) but found 0

❌ python/cohere-v2 :: Basic Error LLM Test (sync, blocking)

Error: 1 check(s) failed:

1 check(s) failed:
Should have at least 1 AI span(s) but found 0

❌ python/cohere-v2 :: Conversation ID LLM Test (sync, streaming)

Error: Test execution failed: Command failed: /home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/.venv/bin/python /home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/test-conversation-id-llm-test-sync-streaming.py

Test execution failed: Command failed: /home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/.venv/bin/python /home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/test-conversation-id-llm-test-sync-streaming.py
Traceback (most recent call last):
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/test-conversation-id-llm-test-sync-streaming.py", line 97, in <module>
    main()
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/test-conversation-id-llm-test-sync-streaming.py", line 51, in main
    for event in stream:
                 ^^^^^^
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/.venv/lib/python3.12/site-packages/cohere/v2/client.py", line 333, in chat_stream
    raise TooManyRequestsError(
cohere.errors.too_many_requests_error.TooManyRequestsError: status_code: 429, body: data=None id='663fe78e-3c86-4d42-853e-6a151ab29052' message="You are using a Trial key, which is limited to 20 API calls / minute. You can continue to use the Trial key for free or upgrade to a Production key with higher rate limits at 'https://dashboard.cohere.com/api-keys'. Contact us on 'https://discord.gg/XW44jPfYJu' or email us at support@cohere.com with any questions"

Traceback (most recent call last):
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/test-conversation-id-llm-test-sync-streaming.py", line 97, in <module>
    main()
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/test-conversation-id-llm-test-sync-streaming.py", line 51, in main
    for event in stream:
                 ^^^^^^
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/.venv/lib/python3.12/site-packages/cohere/v2/client.py", line 333, in chat_stream
    raise TooManyRequestsError(
cohere.errors.too_many_requests_error.TooManyRequestsError: status_code: 429, body: data=None id='663fe78e-3c86-4d42-853e-6a151ab29052' message="You are using a Trial key, which is limited to 20 API calls / minute. You can continue to use the Trial key for free or upgrade to a Production key with higher rate limits at 'https://dashboard.cohere.com/api-keys'. Contact us on 'https://discord.gg/XW44jPfYJu' or email us at support@cohere.com with any questions"

❌ python/cohere-v2 :: Conversation ID LLM Test (sync, blocking)

Error: Test execution failed: Command failed: /home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/.venv/bin/python /home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/test-conversation-id-llm-test-sync-blocking.py

Test execution failed: Command failed: /home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/.venv/bin/python /home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/test-conversation-id-llm-test-sync-blocking.py
Traceback (most recent call last):
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/test-conversation-id-llm-test-sync-blocking.py", line 81, in <module>
    main()
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/test-conversation-id-llm-test-sync-blocking.py", line 42, in main
    response = client.chat(
               ^^^^^^^^^^^^
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/.venv/lib/python3.12/site-packages/cohere/client.py", line 103, in _wrapped
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/.venv/lib/python3.12/site-packages/cohere/client.py", line 35, in _wrapped
    return method(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/.venv/lib/python3.12/site-packages/cohere/v2/client.py", line 636, in chat
    raise TooManyRequestsError(
cohere.errors.too_many_requests_error.TooManyRequestsError: status_code: 429, body: data=None id='7c0e4b22-17dd-48dd-8443-955fdd933271' message="You are using a Trial key, which is limited to 20 API calls / minute. You can continue to use the Trial key for free or upgrade to a Production key with higher rate limits at 'https://dashboard.cohere.com/api-keys'. Contact us on 'https://discord.gg/XW44jPfYJu' or email us at support@cohere.com with any questions"

Traceback (most recent call last):
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/test-conversation-id-llm-test-sync-blocking.py", line 81, in <module>
    main()
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/test-conversation-id-llm-test-sync-blocking.py", line 42, in main
    response = client.chat(
               ^^^^^^^^^^^^
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/.venv/lib/python3.12/site-packages/cohere/client.py", line 103, in _wrapped
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/.venv/lib/python3.12/site-packages/cohere/client.py", line 35, in _wrapped
    return method(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/runs/python/cohere-v2-5.13.0-sentry-latest/.venv/lib/python3.12/site-packages/cohere/v2/client.py", line 636, in chat
    raise TooManyRequestsError(
cohere.errors.too_many_requests_error.TooManyRequestsError: status_code: 429, body: data=None id='7c0e4b22-17dd-48dd-8443-955fdd933271' message="You are using a Trial key, which is limited to 20 API calls / minute. You can continue to use the Trial key for free or upgrade to a Production key with higher rate limits at 'https://dashboard.cohere.com/api-keys'. Contact us on 'https://discord.gg/XW44jPfYJu' or email us at support@cohere.com with any questions"

Test Matrix

Agent Tests

SDK	Basic Agent Test	Conversation ID Agent Test	Long Input Agent Test	Tool Call Agent Test	Tool Error Agent Test	Vision Agent Test
cloudflare/langgraph	❌	❌	❌	❌	❌	❌
cloudflare/vercel	❌	❌	❌	❌	❌	❌
nextjs/mastra	✅	❌	—	✅	✅	✅
nextjs/vercel	❌_blk ❌_str	❌_blk ❌_str	❌_blk ❌_str	❌_blk ❌_str	❌_blk ❌_str	❌_blk ❌_str
node/manual	✅	❌	✅	✅	✅	✅
node/mastra	❌	❌	❌	❌	❌	❌
node/vercel	❌	❌	❌	❌	❌	❌
php/laravel	❌_blk ❌_str	❌_blk ❌_str	❌_blk ❌_str	❌_blk ❌_str	❌_blk ❌_str	❌_blk ❌_str
python/langgraph	❌_a ❌_s	❌_a ❌_s	❌_a ❌_s	❌_a ❌_s	❌_a ❌_s	❌_a ❌_s
python/manual	✅_a ✅_s	✅_a ✅_s	✅_a ✅_s	✅_a ✅_s	✅_a ✅_s	✅_a ✅_s
python/openai-agents	✅	✅	❌	✅	✅	❌
python/pydantic-ai	✅	✅	❌	❌	❌	❌

Embedding Tests

SDK	Basic Embeddings Test
browser/google-genai	❌
browser/langchain	❌
browser/openai	✅
cloudflare/google-genai	❌
cloudflare/langchain	❌
cloudflare/openai	✅
cloudflare/vercel	❌
nextjs/google-genai	❌
nextjs/langchain	❌
nextjs/openai	✅
nextjs/vercel	❌
node/google-genai	❌
node/langchain	❌
node/openai	✅
node/vercel	❌
php/laravel	❌
python/google-genai	❌_{a, blk} ❌_{s, blk}
python/langchain	❌_{a, blk} ❌_{s, blk}
python/litellm	❌_{a, blk} ✅_{s, blk}
python/manual	✅_{a, blk} ✅_{s, blk}
python/openai	✅_{a, blk} ✅_{s, blk}

LLM Tests

SDK	Basic Error LLM Test	Basic LLM Test	Conversation ID LLM Test	Long Input LLM Test	Multi-Turn LLM Test	Vision LLM Test
browser/anthropic	✅_blk ❌_str	✅_blk ✅_str	❌_blk ❌_str	✅_blk ✅_str	✅_blk ✅_str	✅_blk ✅_str
browser/google-genai	✅_blk ✅_str	✅_blk ✅_str	❌_blk ❌_str	❌_blk ❌_str	❌_blk ❌_str	✅_blk ✅_str
browser/langchain	❌_blk ❌_str	❌_blk ❌_str	❌_blk ❌_str	❌_blk ❌_str	❌_blk ❌_str	❌_blk ❌_str
browser/openai	✅_blk ✅_str	✅_blk ❌_str	❌_blk ❌_str	✅_blk ❌_str	✅_blk ❌_str	✅_blk ❌_str
cloudflare/anthropic	✅_blk ❌_str	✅_blk ✅_str	❌_blk ❌_str	✅_blk ✅_str	✅_blk ✅_str	✅_blk ✅_str
cloudflare/google-genai	✅_blk ✅_str	✅_blk ✅_str	❌_blk ❌_str	✅_blk ✅_str	✅_blk ✅_str	✅🔧_blk ✅_str
cloudflare/langchain	❌_blk ❌_str	❌_blk ❌_str	❌_blk ❌_str	❌_blk ❌_str	❌_blk ❌_str	❌_blk ❌_str
cloudflare/openai	✅_blk ✅_str	✅_blk ✅_str	❌_blk ❌_str	✅_blk ✅_str	✅_blk ✅_str	✅_blk ✅_str
nextjs/anthropic	✅_blk ✅_str	❌_blk ❌_str	❌_blk ❌_str	❌_blk ❌_str	❌_blk ❌_str	❌_blk ❌_str
nextjs/google-genai	✅_blk ✅_str	❌_blk ❌_str	❌_blk ❌_str	❌_blk ❌_str	❌_blk ❌_str	❌_blk ❌_str
nextjs/langchain	✅_blk ✅_str	❌_blk ❌_str	❌_blk ❌_str	❌_blk ❌_str	❌_blk ❌_str	❌_blk ❌_str
nextjs/openai	✅_blk ✅_str	✅_blk ❌_str	✅_blk ❌_str	✅_blk ❌_str	✅_blk ❌_str	✅_blk ❌_str
node/anthropic	✅_blk ✅_str	✅_blk ✅_str	✅_blk ✅_str	✅_blk ✅_str	✅_blk ✅_str	✅_blk ✅_str
node/google-genai	✅_blk ✅_str	✅_blk ✅_str	✅_blk ✅_str	✅_blk ✅_str	✅_blk ✅_str	✅_blk ✅_str
node/langchain	✅_blk ✅_str	✅_blk ✅_str	✅_blk ✅_str	✅_blk ✅_str	✅_blk ✅_str	✅_blk ✅_str
node/manual	—	✅	✅	✅	✅	✅
node/openai	✅_blk ✅_str	✅_blk ✅_str	✅_blk ✅_str	✅_blk ✅_str	✅_blk ✅_str	✅_blk ✅_str
python/anthropic	✅_{a, blk} ❌_{a, str} ✅_{s, blk} ❌_{s, str}	✅_{a, blk} ❌_{a, str} ✅_{s, blk} ❌_{s, str}	✅_{a, blk} ❌_{a, str} ✅_{s, blk} ❌_{s, str}	✅_{a, blk} ❌_{a, str} ✅_{s, blk} ❌_{s, str}	✅_{a, blk} ❌_{a, str} ✅_{s, blk} ❌_{s, str}	✅_{a, blk} ❌_{a, str} ✅_{s, blk} ❌_{s, str}
python/cohere	❌🆕_{s, blk} ❌🆕_{s, str}	❌🆕_{s, blk} ❌🆕_{s, str}	❌🆕_{s, blk} ❌🆕_{s, str}	—	❌🆕_{s, blk} ❌🆕_{s, str}	—
python/cohere-v2	❌🆕_{s, blk} ❌🆕_{s, str}	❌🆕_{s, blk} ❌🆕_{s, str}	❌🆕_{s, blk} ❌🆕_{s, str}	—	❌🆕_{s, blk} ❌🆕_{s, str}	—
python/google-genai	✅_{a, blk} ✅_{a, str} ✅_{s, blk} ✅_{s, str}	✅_{a, blk} ✅_{a, str} ✅_{s, blk} ✅_{s, str}	✅_{a, blk} ✅_{a, str} ✅_{s, blk} ✅_{s, str}	✅_{a, blk} ✅_{a, str} ✅_{s, blk} ✅_{s, str}	❌_{a, blk} ❌_{a, str} ❌_{s, blk} ❌_{s, str}	❌_{a, blk} ❌_{a, str} ❌_{s, blk} ❌_{s, str}
python/langchain	❌_{a, blk} ❌_{a, str} ❌_{s, blk} ❌_{s, str}	✅_{a, blk} ✅_{a, str} ✅_{s, blk} ✅_{s, str}	✅_{a, blk} ✅_{a, str} ✅_{s, blk} ✅_{s, str}	✅_{a, blk} ✅_{a, str} ✅_{s, blk} ✅_{s, str}	✅_{a, blk} ✅_{a, str} ✅_{s, blk} ✅_{s, str}	✅_{a, blk} ✅_{a, str} ✅_{s, blk} ✅_{s, str}
python/litellm	❌_{a, blk} ❌_{a, str} ❌_{s, blk} ❌_{s, str}	❌_{a, blk} ❌_{a, str} ✅_{s, blk} ✅_{s, str}	❌_{a, blk} ❌_{a, str} ✅_{s, blk} ✅_{s, str}	❌_{a, blk} ❌_{a, str} ✅_{s, blk} ✅_{s, str}	❌_{a, blk} ❌_{a, str} ✅_{s, blk} ✅_{s, str}	❌_{a, blk} ❌_{a, str} ✅_{s, blk} ✅_{s, str}
python/manual	—	✅_{a, blk} ✅_{s, blk}	✅_{a, blk} ✅_{s, blk}	✅_{a, blk} ✅_{s, blk}	✅_{a, blk} ✅_{s, blk}	✅_{a, blk} ✅_{s, blk}
python/openai	❌_{a, blk} ❌_{a, str} ❌_{s, blk} ❌_{s, str}	✅_{a, blk} ❌_{a, str} ✅_{s, blk} ❌_{s, str}	✅_{a, blk} ❌_{a, str} ✅_{s, blk} ❌_{s, str}	✅_{a, blk} ❌_{a, str} ✅_{s, blk} ❌_{s, str}	✅_{a, blk} ❌_{a, str} ✅_{s, blk} ❌_{s, str}	❌_{a, blk} ❌_{a, str} ❌_{s, blk} ❌_{s, str}

Generated by AI SDK Integration Tests

linear-code · 2026-03-06T07:11:59Z

TET-2025 Add tests for cohere to testing framework

src/runner/templates/llm/python/cohere/config.json

src/runner/templates/llm/python/cohere/template.njk

src/runner/templates/llm/python/cohere-v2/config.json

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

.github/workflows/pr-tests.yml

feat: add tests for cohere

8ca1976

shellmayr requested a review from constantinius March 6, 2026 07:11

shellmayr marked this pull request as ready for review March 6, 2026 07:12

sentry bot reviewed Mar 6, 2026

View reviewed changes

src/runner/templates/llm/python/cohere/config.json Outdated Show resolved Hide resolved

cursor bot reviewed Mar 6, 2026

View reviewed changes

src/runner/templates/llm/python/cohere/config.json Outdated Show resolved Hide resolved

src/runner/templates/llm/python/cohere/template.njk Outdated Show resolved Hide resolved

move py -> python

9192fcb

sentry bot reviewed Mar 9, 2026

View reviewed changes

src/runner/templates/llm/python/cohere-v2/config.json Outdated Show resolved Hide resolved

shellmayr and others added 3 commits March 9, 2026 10:11

fix config

44039f3

fix template

f9a6e4e

fix: passing CO_API_KEY in CI workflows

0b3b283

cursor bot reviewed Mar 9, 2026

View reviewed changes

.github/workflows/pr-tests.yml Show resolved Hide resolved

fix: missing env var passing

bf336cb

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: add tests for cohere#63

feat: add tests for cohere#63
shellmayr wants to merge 6 commits intomainfrom
shellmayr/feat/add-tests-for-cohere

shellmayr commented Feb 27, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Feb 27, 2026 •

edited

Loading

Uh oh!

linear-code bot commented Mar 6, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor bot left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

shellmayr commented Feb 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟡 AI SDK Integration Test Results

Summary

✅ Fixed

🆕 New Tests

Test Matrix

Agent Tests

Embedding Tests

LLM Tests

Uh oh!

linear-code bot commented Mar 6, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

shellmayr commented Feb 27, 2026 •

edited

Loading

github-actions bot commented Feb 27, 2026 •

edited

Loading