From 5c73738b378243ccdf4af108aefe27293aaf05da Mon Sep 17 00:00:00 2001 From: JR Boos Date: Thu, 14 May 2026 09:38:32 -0400 Subject: [PATCH 1/3] (lcore-2079) added agent skills e2e feature file --- tests/e2e/features/skills.feature | 377 ++++++++++++++++++++++++++++++ 1 file changed, 377 insertions(+) create mode 100644 tests/e2e/features/skills.feature diff --git a/tests/e2e/features/skills.feature b/tests/e2e/features/skills.feature new file mode 100644 index 000000000..d41aa84e5 --- /dev/null +++ b/tests/e2e/features/skills.feature @@ -0,0 +1,377 @@ +@e2e_group_2 +Feature: Agent skills tests + + Background: + Given The service is started locally + And The system is in default state + And I set the Authorization header to Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6Ikpva + And REST API service prefix is /v1 + And the Lightspeed stack configuration directory is "tests/e2e/configuration" + + # --- Configuration & startup --- + + @SkillsConfig + Scenario: Service starts successfully with skills configured + Given The e2e-test-skill skill directory is present in the container + And The service uses the lightspeed-stack-skills.yaml configuration + And The service is restarted + When I access endpoint "readiness" using HTTP GET method + Then The status code of the response is 200 + And The body of the response contains true + + @SkillsConfig + Scenario: Service fails to start when skill path does not exist + Given The service uses the lightspeed-stack-skills-invalid-path.yaml configuration + And The service is restarted + When I access endpoint "readiness" using HTTP GET method + Then The status code of the response is not 200 + + @SkillsConfig + Scenario: Service fails to start when SKILL.md is missing from skill directory + Given The e2e-missing-skillmd-skill skill directory is present in the container + And The service uses the lightspeed-stack-skills-missing-skillmd.yaml configuration + And The service is restarted + When I access endpoint "readiness" using HTTP GET method + Then The status code of the response is not 200 + + @SkillsConfig + Scenario: Service fails to start when SKILL.md has invalid frontmatter + Given The service uses the lightspeed-stack-skills-invalid-frontmatter.yaml configuration + And The service is restarted + When I access endpoint "readiness" using HTTP GET method + Then The status code of the response is not 200 + + @SkillsConfig + Scenario: Service fails to start when duplicate skill names are configured + Given The service uses the lightspeed-stack-skills-duplicate-names.yaml configuration + And The service is restarted + When I access endpoint "readiness" using HTTP GET method + Then The status code of the response is not 200 + + + # --- Skill tools registration --- + + @SkillsConfig + Scenario: Skill tools are registered when skills are configured + Given The e2e-test-skill skill directory is present in the container + And The service uses the lightspeed-stack-skills.yaml configuration + And The service is restarted + When I access REST API endpoint "tools" using HTTP GET method + Then The status code of the response is 200 + And The body of the response contains list_skills + And The body of the response contains activate_skill + And The body of the response contains load_skill_resource + + Scenario: Skill tools are not registered when no skills are configured + Given The service uses the lightspeed-stack.yaml configuration + And The service is restarted + When I access REST API endpoint "tools" using HTTP GET method + Then The status code of the response is 200 + And The body of the response does not contain list_skills + And The body of the response does not contain activate_skill + And The body of the response does not contain load_skill_resource + + + # --- Skill discovery --- + + @SkillsConfig @Authorized @flaky + Scenario: LLM can discover skills via list_skills tool using query endpoint + Given The e2e-test-skill skill directory is present in the container + And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration + And The service is restarted + And I capture the current token metrics + When I use "query" to ask question with authorization header + """ + {"query": "What skills are available? Use the list_skills tool.", "model": "{MODEL}", "provider": "{PROVIDER}"} + """ + Then The status code of the response is 200 + And The body of the response contains e2e-test-skill + And The token metrics have increased + + @SkillsConfig @Authorized @flaky + Scenario: LLM can discover skills via list_skills tool using streaming_query endpoint + Given The e2e-test-skill skill directory is present in the container + And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration + And The service is restarted + And I capture the current token metrics + When I use "streaming_query" to ask question with authorization header + """ + {"query": "What skills are available? Use the list_skills tool.", "model": "{MODEL}", "provider": "{PROVIDER}"} + """ + When I wait for the response to be completed + Then The status code of the response is 200 + And The streamed response contains following fragments + | Fragments in LLM response | + | e2e-test-skill | + And The token metrics have increased + + + # --- Skill activation --- + + @SkillsConfig @Authorized @flaky + Scenario: LLM can activate a skill and use its instructions via query endpoint + Given The e2e-test-skill skill directory is present in the container + And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration + And The service is restarted + And I capture the current token metrics + When I use "query" to ask question with authorization header + """ + {"query": "I need help with e2e testing. Use the activate_skill tool to load the e2e-test-skill.", "model": "{MODEL}", "provider": "{PROVIDER}"} + """ + Then The status code of the response is 200 + And The body of the response contains skill_content + And The token metrics have increased + + @SkillsConfig @Authorized @flaky + Scenario: LLM can activate a skill and use its instructions via streaming_query endpoint + Given The e2e-test-skill skill directory is present in the container + And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration + And The service is restarted + And I capture the current token metrics + When I use "streaming_query" to ask question with authorization header + """ + {"query": "I need help with e2e testing. Use the activate_skill tool to load the e2e-test-skill.", "model": "{MODEL}", "provider": "{PROVIDER}"} + """ + When I wait for the response to be completed + Then The status code of the response is 200 + And The streamed response contains following fragments + | Fragments in LLM response | + | skill_content | + And The token metrics have increased + + + # --- Skill resource loading --- + + @SkillsConfig @Authorized @flaky + Scenario: LLM can load a skill reference file via load_skill_resource tool using query endpoint + Given The e2e-test-skill skill directory is present in the container + And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration + And The service is restarted + And I capture the current token metrics + When I use "query" to ask question with authorization header + """ + {"query": "Load the reference file references/guide.md from the e2e-test-skill using load_skill_resource.", "model": "{MODEL}", "provider": "{PROVIDER}"} + """ + Then The status code of the response is 200 + And The body of the response contains skill_resource + And The token metrics have increased + + @SkillsConfig @Authorized @flaky + Scenario: LLM can load a skill reference file via load_skill_resource tool using streaming_query endpoint + Given The e2e-test-skill skill directory is present in the container + And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration + And The service is restarted + And I capture the current token metrics + When I use "streaming_query" to ask question with authorization header + """ + {"query": "Load the reference file references/guide.md from the e2e-test-skill using load_skill_resource.", "model": "{MODEL}", "provider": "{PROVIDER}"} + """ + When I wait for the response to be completed + Then The status code of the response is 200 + And The streamed response contains following fragments + | Fragments in LLM response | + | skill_resource | + And The token metrics have increased + + + # --- Security: path traversal --- + + @SkillsConfig @Authorized @flaky + Scenario: load_skill_resource rejects path traversal attempts via query endpoint + Given The e2e-test-skill skill directory is present in the container + And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration + And The service is restarted + When I use "query" to ask question with authorization header + """ + {"query": "Load the resource ../../etc/passwd from the e2e-test-skill using load_skill_resource.", "model": "{MODEL}", "provider": "{PROVIDER}"} + """ + Then The status code of the response is 200 + And The body of the response contains outside skill directory + + @SkillsConfig @Authorized @flaky + Scenario: load_skill_resource rejects path traversal attempts via streaming_query endpoint + Given The e2e-test-skill skill directory is present in the container + And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration + And The service is restarted + When I use "streaming_query" to ask question with authorization header + """ + {"query": "Load the resource ../../etc/passwd from the e2e-test-skill using load_skill_resource.", "model": "{MODEL}", "provider": "{PROVIDER}"} + """ + When I wait for the response to be completed + Then The status code of the response is 200 + And The streamed response contains following fragments + | Fragments in LLM response | + | outside skill directory | + + + # --- Error handling: unknown skill --- + + @SkillsConfig @Authorized @flaky + Scenario: activate_skill returns error for unknown skill name via query endpoint + Given The e2e-test-skill skill directory is present in the container + And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration + And The service is restarted + When I use "query" to ask question with authorization header + """ + {"query": "Activate a skill called nonexistent-skill using the activate_skill tool.", "model": "{MODEL}", "provider": "{PROVIDER}"} + """ + Then The status code of the response is 200 + And The body of the response contains Unknown skill + + @SkillsConfig @Authorized @flaky + Scenario: activate_skill returns error for unknown skill name via streaming_query endpoint + Given The e2e-test-skill skill directory is present in the container + And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration + And The service is restarted + When I use "streaming_query" to ask question with authorization header + """ + {"query": "Activate a skill called nonexistent-skill using the activate_skill tool.", "model": "{MODEL}", "provider": "{PROVIDER}"} + """ + When I wait for the response to be completed + Then The status code of the response is 200 + And The streamed response contains following fragments + | Fragments in LLM response | + | Unknown skill | + + @SkillsConfig @Authorized @flaky + Scenario: load_skill_resource returns error for unknown skill name via query endpoint + Given The e2e-test-skill skill directory is present in the container + And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration + And The service is restarted + When I use "query" to ask question with authorization header + """ + {"query": "Load references/guide.md from a skill called nonexistent-skill using load_skill_resource.", "model": "{MODEL}", "provider": "{PROVIDER}"} + """ + Then The status code of the response is 200 + And The body of the response contains Unknown skill + + @SkillsConfig @Authorized @flaky + Scenario: load_skill_resource returns error for unknown skill name via streaming_query endpoint + Given The e2e-test-skill skill directory is present in the container + And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration + And The service is restarted + When I use "streaming_query" to ask question with authorization header + """ + {"query": "Load references/guide.md from a skill called nonexistent-skill using load_skill_resource.", "model": "{MODEL}", "provider": "{PROVIDER}"} + """ + When I wait for the response to be completed + Then The status code of the response is 200 + And The streamed response contains following fragments + | Fragments in LLM response | + | Unknown skill | + + + # --- Error handling: missing resource --- + + @SkillsConfig @Authorized @flaky + Scenario: load_skill_resource returns error for nonexistent resource file via query endpoint + Given The e2e-test-skill skill directory is present in the container + And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration + And The service is restarted + When I use "query" to ask question with authorization header + """ + {"query": "Load references/nonexistent.md from e2e-test-skill using load_skill_resource.", "model": "{MODEL}", "provider": "{PROVIDER}"} + """ + Then The status code of the response is 200 + And The body of the response contains Resource not found + + @SkillsConfig @Authorized @flaky + Scenario: load_skill_resource returns error for nonexistent resource file via streaming_query endpoint + Given The e2e-test-skill skill directory is present in the container + And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration + And The service is restarted + When I use "streaming_query" to ask question with authorization header + """ + {"query": "Load references/nonexistent.md from e2e-test-skill using load_skill_resource.", "model": "{MODEL}", "provider": "{PROVIDER}"} + """ + When I wait for the response to be completed + Then The status code of the response is 200 + And The streamed response contains following fragments + | Fragments in LLM response | + | Resource not found | + + + # --- Context management: deduplication --- + + @SkillsConfig @Authorized @flaky + Scenario: Duplicate skill activation in same conversation returns already-loaded note via query endpoint + Given The e2e-test-skill skill directory is present in the container + And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration + And The service is restarted + When I use "query" to ask question with authorization header + """ + {"query": "Activate e2e-test-skill using the activate_skill tool.", "model": "{MODEL}", "provider": "{PROVIDER}"} + """ + Then The status code of the response is 200 + And I store conversation details + When I use "query" to ask question with same conversation_id + """ + {"query": "Activate e2e-test-skill again using the activate_skill tool.", "model": "{MODEL}", "provider": "{PROVIDER}"} + """ + Then The status code of the response is 200 + And The body of the response contains already loaded + + + # --- Multiple skills --- + + @SkillsMultiConfig @Authorized @flaky + Scenario: Multiple skills can be discovered via query endpoint + Given The e2e-test-skill skill directory is present in the container + And The e2e-second-skill skill directory is present in the container + And The service uses the lightspeed-stack-skills-multi.yaml configuration + And The service is restarted + When I use "query" to ask question with authorization header + """ + {"query": "List all available skills using the list_skills tool.", "model": "{MODEL}", "provider": "{PROVIDER}"} + """ + Then The status code of the response is 200 + And The body of the response contains e2e-test-skill + And The body of the response contains e2e-second-skill + + @SkillsMultiConfig @Authorized @flaky + Scenario: Multiple skills can be discovered via streaming_query endpoint + Given The e2e-test-skill skill directory is present in the container + And The e2e-second-skill skill directory is present in the container + And The service uses the lightspeed-stack-skills-multi.yaml configuration + And The service is restarted + When I use "streaming_query" to ask question with authorization header + """ + {"query": "List all available skills using the list_skills tool.", "model": "{MODEL}", "provider": "{PROVIDER}"} + """ + When I wait for the response to be completed + Then The status code of the response is 200 + And The streamed response contains following fragments + | Fragments in LLM response | + | e2e-test-skill | + | e2e-second-skill | + + @SkillsMultiConfig @Authorized @flaky + Scenario: Skills directory path discovers all skills in subdirectories via query endpoint + Given The e2e-test-skill skill directory is present in the container + And The e2e-second-skill skill directory is present in the container + And The service uses the lightspeed-stack-skills-directory.yaml configuration + And The service is restarted + When I use "query" to ask question with authorization header + """ + {"query": "List all available skills using the list_skills tool.", "model": "{MODEL}", "provider": "{PROVIDER}"} + """ + Then The status code of the response is 200 + And The body of the response contains e2e-test-skill + And The body of the response contains e2e-second-skill + + @SkillsMultiConfig @Authorized @flaky + Scenario: Skills directory path discovers all skills in subdirectories via streaming_query endpoint + Given The e2e-test-skill skill directory is present in the container + And The e2e-second-skill skill directory is present in the container + And The service uses the lightspeed-stack-skills-directory.yaml configuration + And The service is restarted + When I use "streaming_query" to ask question with authorization header + """ + {"query": "List all available skills using the list_skills tool.", "model": "{MODEL}", "provider": "{PROVIDER}"} + """ + When I wait for the response to be completed + Then The status code of the response is 200 + And The streamed response contains following fragments + | Fragments in LLM response | + | e2e-test-skill | + | e2e-second-skill | \ No newline at end of file From cdfcf62eeef33ae16cf18f6da44a61ac5db6c7c3 Mon Sep 17 00:00:00 2001 From: JR Boos Date: Fri, 15 May 2026 12:03:29 -0400 Subject: [PATCH 2/3] tmp --- tests/e2e/features/skills.feature | 203 ++++++------------------------ 1 file changed, 39 insertions(+), 164 deletions(-) diff --git a/tests/e2e/features/skills.feature b/tests/e2e/features/skills.feature index d41aa84e5..8be55e32d 100644 --- a/tests/e2e/features/skills.feature +++ b/tests/e2e/features/skills.feature @@ -1,53 +1,14 @@ -@e2e_group_2 +@e2e_group_2 @skip Feature: Agent skills tests Background: Given The service is started locally And The system is in default state - And I set the Authorization header to Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6Ikpva And REST API service prefix is /v1 And the Lightspeed stack configuration directory is "tests/e2e/configuration" - # --- Configuration & startup --- - - @SkillsConfig - Scenario: Service starts successfully with skills configured - Given The e2e-test-skill skill directory is present in the container - And The service uses the lightspeed-stack-skills.yaml configuration - And The service is restarted - When I access endpoint "readiness" using HTTP GET method - Then The status code of the response is 200 - And The body of the response contains true - - @SkillsConfig - Scenario: Service fails to start when skill path does not exist - Given The service uses the lightspeed-stack-skills-invalid-path.yaml configuration - And The service is restarted - When I access endpoint "readiness" using HTTP GET method - Then The status code of the response is not 200 - - @SkillsConfig - Scenario: Service fails to start when SKILL.md is missing from skill directory - Given The e2e-missing-skillmd-skill skill directory is present in the container - And The service uses the lightspeed-stack-skills-missing-skillmd.yaml configuration - And The service is restarted - When I access endpoint "readiness" using HTTP GET method - Then The status code of the response is not 200 - - @SkillsConfig - Scenario: Service fails to start when SKILL.md has invalid frontmatter - Given The service uses the lightspeed-stack-skills-invalid-frontmatter.yaml configuration - And The service is restarted - When I access endpoint "readiness" using HTTP GET method - Then The status code of the response is not 200 - - @SkillsConfig - Scenario: Service fails to start when duplicate skill names are configured - Given The service uses the lightspeed-stack-skills-duplicate-names.yaml configuration - And The service is restarted - When I access endpoint "readiness" using HTTP GET method - Then The status code of the response is not 200 + #TODO: Remove "The e2e-test-skill skill directory is present in the container" # --- Skill tools registration --- @@ -61,6 +22,7 @@ Feature: Agent skills tests And The body of the response contains list_skills And The body of the response contains activate_skill And The body of the response contains load_skill_resource + #TODO: list all the tools, check for number of tools (total) (More comprehensive than just basic testing is +) Scenario: Skill tools are not registered when no skills are configured Given The service uses the lightspeed-stack.yaml configuration @@ -70,31 +32,33 @@ Feature: Agent skills tests And The body of the response does not contain list_skills And The body of the response does not contain activate_skill And The body of the response does not contain load_skill_resource + #TODO: list all the tools, check for number of tools (total should be just non-skill tools) # --- Skill discovery --- - @SkillsConfig @Authorized @flaky + @SkillsConfig Scenario: LLM can discover skills via list_skills tool using query endpoint Given The e2e-test-skill skill directory is present in the container And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration And The service is restarted And I capture the current token metrics - When I use "query" to ask question with authorization header + When I use "query" to ask question """ {"query": "What skills are available? Use the list_skills tool.", "model": "{MODEL}", "provider": "{PROVIDER}"} """ Then The status code of the response is 200 - And The body of the response contains e2e-test-skill + And The body of the response contains e2e-test-skill #TODO: Make this more specific (instead of checking entire response check the skill content metadata (new step required)) + #TODO: Instead of ^ Check tool results from list_skills tool (make it more decisive) And The token metrics have increased - @SkillsConfig @Authorized @flaky + @SkillsConfig Scenario: LLM can discover skills via list_skills tool using streaming_query endpoint Given The e2e-test-skill skill directory is present in the container And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration And The service is restarted And I capture the current token metrics - When I use "streaming_query" to ask question with authorization header + When I use "streaming_query" to ask question """ {"query": "What skills are available? Use the list_skills tool.", "model": "{MODEL}", "provider": "{PROVIDER}"} """ @@ -104,31 +68,32 @@ Feature: Agent skills tests | Fragments in LLM response | | e2e-test-skill | And The token metrics have increased + #TODO: SEE ABOVE TEST # --- Skill activation --- - @SkillsConfig @Authorized @flaky + @SkillsConfig Scenario: LLM can activate a skill and use its instructions via query endpoint Given The e2e-test-skill skill directory is present in the container And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration And The service is restarted And I capture the current token metrics - When I use "query" to ask question with authorization header + When I use "query" to ask question """ {"query": "I need help with e2e testing. Use the activate_skill tool to load the e2e-test-skill.", "model": "{MODEL}", "provider": "{PROVIDER}"} """ Then The status code of the response is 200 - And The body of the response contains skill_content + And The body of the response contains skill_content #FIX: VERY GENERAL should check tool_results instead And The token metrics have increased - @SkillsConfig @Authorized @flaky + @SkillsConfig Scenario: LLM can activate a skill and use its instructions via streaming_query endpoint Given The e2e-test-skill skill directory is present in the container And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration And The service is restarted And I capture the current token metrics - When I use "streaming_query" to ask question with authorization header + When I use "streaming_query" to ask question """ {"query": "I need help with e2e testing. Use the activate_skill tool to load the e2e-test-skill.", "model": "{MODEL}", "provider": "{PROVIDER}"} """ @@ -142,13 +107,13 @@ Feature: Agent skills tests # --- Skill resource loading --- - @SkillsConfig @Authorized @flaky + @SkillsConfig Scenario: LLM can load a skill reference file via load_skill_resource tool using query endpoint Given The e2e-test-skill skill directory is present in the container And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration And The service is restarted And I capture the current token metrics - When I use "query" to ask question with authorization header + When I use "query" to ask question """ {"query": "Load the reference file references/guide.md from the e2e-test-skill using load_skill_resource.", "model": "{MODEL}", "provider": "{PROVIDER}"} """ @@ -156,13 +121,13 @@ Feature: Agent skills tests And The body of the response contains skill_resource And The token metrics have increased - @SkillsConfig @Authorized @flaky + @SkillsConfig Scenario: LLM can load a skill reference file via load_skill_resource tool using streaming_query endpoint Given The e2e-test-skill skill directory is present in the container And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration And The service is restarted And I capture the current token metrics - When I use "streaming_query" to ask question with authorization header + When I use "streaming_query" to ask question """ {"query": "Load the reference file references/guide.md from the e2e-test-skill using load_skill_resource.", "model": "{MODEL}", "provider": "{PROVIDER}"} """ @@ -173,57 +138,26 @@ Feature: Agent skills tests | skill_resource | And The token metrics have increased - - # --- Security: path traversal --- - - @SkillsConfig @Authorized @flaky - Scenario: load_skill_resource rejects path traversal attempts via query endpoint - Given The e2e-test-skill skill directory is present in the container - And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration - And The service is restarted - When I use "query" to ask question with authorization header - """ - {"query": "Load the resource ../../etc/passwd from the e2e-test-skill using load_skill_resource.", "model": "{MODEL}", "provider": "{PROVIDER}"} - """ - Then The status code of the response is 200 - And The body of the response contains outside skill directory - - @SkillsConfig @Authorized @flaky - Scenario: load_skill_resource rejects path traversal attempts via streaming_query endpoint - Given The e2e-test-skill skill directory is present in the container - And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration - And The service is restarted - When I use "streaming_query" to ask question with authorization header - """ - {"query": "Load the resource ../../etc/passwd from the e2e-test-skill using load_skill_resource.", "model": "{MODEL}", "provider": "{PROVIDER}"} - """ - When I wait for the response to be completed - Then The status code of the response is 200 - And The streamed response contains following fragments - | Fragments in LLM response | - | outside skill directory | - - # --- Error handling: unknown skill --- - @SkillsConfig @Authorized @flaky + @SkillsConfig Scenario: activate_skill returns error for unknown skill name via query endpoint Given The e2e-test-skill skill directory is present in the container And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration And The service is restarted - When I use "query" to ask question with authorization header + When I use "query" to ask question """ {"query": "Activate a skill called nonexistent-skill using the activate_skill tool.", "model": "{MODEL}", "provider": "{PROVIDER}"} """ Then The status code of the response is 200 - And The body of the response contains Unknown skill + And The body of the response contains Unknown skill #TODO: Make more descriptive - @SkillsConfig @Authorized @flaky + @SkillsConfig Scenario: activate_skill returns error for unknown skill name via streaming_query endpoint Given The e2e-test-skill skill directory is present in the container And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration And The service is restarted - When I use "streaming_query" to ask question with authorization header + When I use "streaming_query" to ask question """ {"query": "Activate a skill called nonexistent-skill using the activate_skill tool.", "model": "{MODEL}", "provider": "{PROVIDER}"} """ @@ -231,56 +165,28 @@ Feature: Agent skills tests Then The status code of the response is 200 And The streamed response contains following fragments | Fragments in LLM response | - | Unknown skill | - - @SkillsConfig @Authorized @flaky - Scenario: load_skill_resource returns error for unknown skill name via query endpoint - Given The e2e-test-skill skill directory is present in the container - And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration - And The service is restarted - When I use "query" to ask question with authorization header - """ - {"query": "Load references/guide.md from a skill called nonexistent-skill using load_skill_resource.", "model": "{MODEL}", "provider": "{PROVIDER}"} - """ - Then The status code of the response is 200 - And The body of the response contains Unknown skill - - @SkillsConfig @Authorized @flaky - Scenario: load_skill_resource returns error for unknown skill name via streaming_query endpoint - Given The e2e-test-skill skill directory is present in the container - And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration - And The service is restarted - When I use "streaming_query" to ask question with authorization header - """ - {"query": "Load references/guide.md from a skill called nonexistent-skill using load_skill_resource.", "model": "{MODEL}", "provider": "{PROVIDER}"} - """ - When I wait for the response to be completed - Then The status code of the response is 200 - And The streamed response contains following fragments - | Fragments in LLM response | - | Unknown skill | - + | Unknown skill | #TODO: Make descriptive # --- Error handling: missing resource --- - @SkillsConfig @Authorized @flaky + @SkillsConfig Scenario: load_skill_resource returns error for nonexistent resource file via query endpoint Given The e2e-test-skill skill directory is present in the container And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration And The service is restarted - When I use "query" to ask question with authorization header + When I use "query" to ask question """ {"query": "Load references/nonexistent.md from e2e-test-skill using load_skill_resource.", "model": "{MODEL}", "provider": "{PROVIDER}"} """ Then The status code of the response is 200 And The body of the response contains Resource not found - @SkillsConfig @Authorized @flaky + @SkillsConfig Scenario: load_skill_resource returns error for nonexistent resource file via streaming_query endpoint Given The e2e-test-skill skill directory is present in the container And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration And The service is restarted - When I use "streaming_query" to ask question with authorization header + When I use "streaming_query" to ask question """ {"query": "Load references/nonexistent.md from e2e-test-skill using load_skill_resource.", "model": "{MODEL}", "provider": "{PROVIDER}"} """ @@ -293,12 +199,12 @@ Feature: Agent skills tests # --- Context management: deduplication --- - @SkillsConfig @Authorized @flaky + @SkillsConfig Scenario: Duplicate skill activation in same conversation returns already-loaded note via query endpoint - Given The e2e-test-skill skill directory is present in the container + Given The e2e-test-skill skill directory path is "e2e-test-skill" And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration And The service is restarted - When I use "query" to ask question with authorization header + When I use "query" to ask question """ {"query": "Activate e2e-test-skill using the activate_skill tool.", "model": "{MODEL}", "provider": "{PROVIDER}"} """ @@ -309,49 +215,18 @@ Feature: Agent skills tests {"query": "Activate e2e-test-skill again using the activate_skill tool.", "model": "{MODEL}", "provider": "{PROVIDER}"} """ Then The status code of the response is 200 - And The body of the response contains already loaded + And The body of the response contains already loaded #FIX: This looks/feels wrong (make more descriptve) # --- Multiple skills --- - @SkillsMultiConfig @Authorized @flaky - Scenario: Multiple skills can be discovered via query endpoint - Given The e2e-test-skill skill directory is present in the container - And The e2e-second-skill skill directory is present in the container - And The service uses the lightspeed-stack-skills-multi.yaml configuration - And The service is restarted - When I use "query" to ask question with authorization header - """ - {"query": "List all available skills using the list_skills tool.", "model": "{MODEL}", "provider": "{PROVIDER}"} - """ - Then The status code of the response is 200 - And The body of the response contains e2e-test-skill - And The body of the response contains e2e-second-skill - - @SkillsMultiConfig @Authorized @flaky - Scenario: Multiple skills can be discovered via streaming_query endpoint - Given The e2e-test-skill skill directory is present in the container - And The e2e-second-skill skill directory is present in the container - And The service uses the lightspeed-stack-skills-multi.yaml configuration - And The service is restarted - When I use "streaming_query" to ask question with authorization header - """ - {"query": "List all available skills using the list_skills tool.", "model": "{MODEL}", "provider": "{PROVIDER}"} - """ - When I wait for the response to be completed - Then The status code of the response is 200 - And The streamed response contains following fragments - | Fragments in LLM response | - | e2e-test-skill | - | e2e-second-skill | - - @SkillsMultiConfig @Authorized @flaky + @SkillsMultiConfig Scenario: Skills directory path discovers all skills in subdirectories via query endpoint Given The e2e-test-skill skill directory is present in the container And The e2e-second-skill skill directory is present in the container And The service uses the lightspeed-stack-skills-directory.yaml configuration And The service is restarted - When I use "query" to ask question with authorization header + When I use "query" to ask question """ {"query": "List all available skills using the list_skills tool.", "model": "{MODEL}", "provider": "{PROVIDER}"} """ @@ -359,13 +234,13 @@ Feature: Agent skills tests And The body of the response contains e2e-test-skill And The body of the response contains e2e-second-skill - @SkillsMultiConfig @Authorized @flaky + @SkillsMultiConfig Scenario: Skills directory path discovers all skills in subdirectories via streaming_query endpoint - Given The e2e-test-skill skill directory is present in the container + Given The e2e-test-skill skill directory path is "skills/e2e-test-skill" And The e2e-second-skill skill directory is present in the container And The service uses the lightspeed-stack-skills-directory.yaml configuration And The service is restarted - When I use "streaming_query" to ask question with authorization header + When I use "streaming_query" to ask question """ {"query": "List all available skills using the list_skills tool.", "model": "{MODEL}", "provider": "{PROVIDER}"} """ From a0789307dedae4dc7d3c71ec1482dff2082a2ea1 Mon Sep 17 00:00:00 2001 From: JR Boos Date: Tue, 19 May 2026 15:41:07 -0400 Subject: [PATCH 3/3] fix --- tests/e2e/features/skills.feature | 296 ++++++++++++++++++++++++------ 1 file changed, 238 insertions(+), 58 deletions(-) diff --git a/tests/e2e/features/skills.feature b/tests/e2e/features/skills.feature index 8be55e32d..2ddcfaf7c 100644 --- a/tests/e2e/features/skills.feature +++ b/tests/e2e/features/skills.feature @@ -7,39 +7,75 @@ Feature: Agent skills tests And REST API service prefix is /v1 And the Lightspeed stack configuration directory is "tests/e2e/configuration" - - #TODO: Remove "The e2e-test-skill skill directory is present in the container" - # --- Skill tools registration --- @SkillsConfig Scenario: Skill tools are registered when skills are configured - Given The e2e-test-skill skill directory is present in the container + Given The e2e-test-skill skill directory path is "e2e-test-skill" And The service uses the lightspeed-stack-skills.yaml configuration And The service is restarted When I access REST API endpoint "tools" using HTTP GET method Then The status code of the response is 200 - And The body of the response contains list_skills - And The body of the response contains activate_skill - And The body of the response contains load_skill_resource - #TODO: list all the tools, check for number of tools (total) (More comprehensive than just basic testing is +) + And The body of the response is the following #TODO: Currently placeholder, should reflect actual tools (all tools not just skill tools) + """ + { + "tools": [ + { + "identifier": "filesystem_read", + "description": "Read contents of a file from the filesystem", + "parameters": [ + { + "name": "path", + "description": "Path to the file to read", + "parameter_type": "string", + "required": True, + "default": None, + } + ], + "provider_id": "model-context-protocol", + "toolgroup_id": "filesystem-tools", + "server_source": "http://localhost:3000", + "type": "tool", + } + ], + } + """ Scenario: Skill tools are not registered when no skills are configured Given The service uses the lightspeed-stack.yaml configuration And The service is restarted When I access REST API endpoint "tools" using HTTP GET method Then The status code of the response is 200 - And The body of the response does not contain list_skills - And The body of the response does not contain activate_skill - And The body of the response does not contain load_skill_resource - #TODO: list all the tools, check for number of tools (total should be just non-skill tools) - + And The body of the response is the following #TODO: Currently placeholder, should reflect actual tools (default tools, not skill tools) + """ + { + "tools": [ + { + "identifier": "filesystem_read", + "description": "Read contents of a file from the filesystem", + "parameters": [ + { + "name": "path", + "description": "Path to the file to read", + "parameter_type": "string", + "required": True, + "default": None, + } + ], + "provider_id": "model-context-protocol", + "toolgroup_id": "filesystem-tools", + "server_source": "http://localhost:3000", + "type": "tool", + } + ], + } + """ # --- Skill discovery --- @SkillsConfig Scenario: LLM can discover skills via list_skills tool using query endpoint - Given The e2e-test-skill skill directory is present in the container + Given The e2e-test-skill skill directory path is "e2e-test-skill" And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration And The service is restarted And I capture the current token metrics @@ -48,13 +84,23 @@ Feature: Agent skills tests {"query": "What skills are available? Use the list_skills tool.", "model": "{MODEL}", "provider": "{PROVIDER}"} """ Then The status code of the response is 200 - And The body of the response contains e2e-test-skill #TODO: Make this more specific (instead of checking entire response check the skill content metadata (new step required)) - #TODO: Instead of ^ Check tool results from list_skills tool (make it more decisive) + And The body of the "tool_results" field is #TODO: Currently placeholder, should reflect actual tool results + """ + [ + { + "id": "1", + "status": "success", + "content": "bla", + "type": "tool_result", + "round": 1, + } + ] + """ And The token metrics have increased @SkillsConfig Scenario: LLM can discover skills via list_skills tool using streaming_query endpoint - Given The e2e-test-skill skill directory is present in the container + Given The e2e-test-skill skill directory path is "e2e-test-skill" And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration And The service is restarted And I capture the current token metrics @@ -64,18 +110,26 @@ Feature: Agent skills tests """ When I wait for the response to be completed Then The status code of the response is 200 - And The streamed response contains following fragments - | Fragments in LLM response | - | e2e-test-skill | + And The response is the last streamed fragment + And The body of the "tool_results" field is #TODO: Currently placeholder, should reflect actual tool results + """ + [ + { + "id": "1", + "status": "success", + "content": "bla", + "type": "tool_result", + "round": 1, + } + ] + """ And The token metrics have increased - #TODO: SEE ABOVE TEST - # --- Skill activation --- @SkillsConfig Scenario: LLM can activate a skill and use its instructions via query endpoint - Given The e2e-test-skill skill directory is present in the container + Given The e2e-test-skill skill directory path is "e2e-test-skill" And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration And The service is restarted And I capture the current token metrics @@ -84,12 +138,23 @@ Feature: Agent skills tests {"query": "I need help with e2e testing. Use the activate_skill tool to load the e2e-test-skill.", "model": "{MODEL}", "provider": "{PROVIDER}"} """ Then The status code of the response is 200 - And The body of the response contains skill_content #FIX: VERY GENERAL should check tool_results instead + And The body of the "tool_results" field is #TODO: Currently placeholder, should reflect actual tool results + """ + [ + { + "id": "1", + "status": "success", + "content": "bla", + "type": "tool_result", + "round": 1, + } + ] + """ And The token metrics have increased @SkillsConfig Scenario: LLM can activate a skill and use its instructions via streaming_query endpoint - Given The e2e-test-skill skill directory is present in the container + Given The e2e-test-skill skill directory path is "e2e-test-skill" And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration And The service is restarted And I capture the current token metrics @@ -99,9 +164,19 @@ Feature: Agent skills tests """ When I wait for the response to be completed Then The status code of the response is 200 - And The streamed response contains following fragments - | Fragments in LLM response | - | skill_content | + And The response is the last streamed fragment + And The body of the "tool_results" field is #TODO: Currently placeholder, should reflect actual tool results + """ + [ + { + "id": "1", + "status": "success", + "content": "bla", + "type": "tool_result", + "round": 1, + } + ] + """ And The token metrics have increased @@ -109,7 +184,7 @@ Feature: Agent skills tests @SkillsConfig Scenario: LLM can load a skill reference file via load_skill_resource tool using query endpoint - Given The e2e-test-skill skill directory is present in the container + Given The e2e-test-skill skill directory path is "e2e-test-skill" And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration And The service is restarted And I capture the current token metrics @@ -118,12 +193,23 @@ Feature: Agent skills tests {"query": "Load the reference file references/guide.md from the e2e-test-skill using load_skill_resource.", "model": "{MODEL}", "provider": "{PROVIDER}"} """ Then The status code of the response is 200 - And The body of the response contains skill_resource + And The body of the "tool_results" field is #TODO: Currently placeholder, should reflect actual tool results + """ + [ + { + "id": "1", + "status": "success", + "content": "bla", + "type": "tool_result", + "round": 1, + } + ] + """ And The token metrics have increased @SkillsConfig Scenario: LLM can load a skill reference file via load_skill_resource tool using streaming_query endpoint - Given The e2e-test-skill skill directory is present in the container + Given The e2e-test-skill skill directory path is "e2e-test-skill" And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration And The service is restarted And I capture the current token metrics @@ -133,16 +219,26 @@ Feature: Agent skills tests """ When I wait for the response to be completed Then The status code of the response is 200 - And The streamed response contains following fragments - | Fragments in LLM response | - | skill_resource | + And The response is the last streamed fragment + And The body of the "tool_results" field is #TODO: Currently placeholder, should reflect actual tool results + """ + [ + { + "id": "1", + "status": "success", + "content": "bla", + "type": "tool_result", + "round": 1, + } + ] + """ And The token metrics have increased # --- Error handling: unknown skill --- @SkillsConfig Scenario: activate_skill returns error for unknown skill name via query endpoint - Given The e2e-test-skill skill directory is present in the container + Given The e2e-test-skill skill directory path is "e2e-test-skill" And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration And The service is restarted When I use "query" to ask question @@ -150,11 +246,22 @@ Feature: Agent skills tests {"query": "Activate a skill called nonexistent-skill using the activate_skill tool.", "model": "{MODEL}", "provider": "{PROVIDER}"} """ Then The status code of the response is 200 - And The body of the response contains Unknown skill #TODO: Make more descriptive + And The body of the "tool_results" field is #TODO: Currently placeholder, should reflect actual tool results + """ + [ + { + "id": "1", + "status": "failure", + "content": "bla", + "type": "tool_result", + "round": 1, + } + ] + """ @SkillsConfig Scenario: activate_skill returns error for unknown skill name via streaming_query endpoint - Given The e2e-test-skill skill directory is present in the container + Given The e2e-test-skill skill directory path is "e2e-test-skill" And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration And The service is restarted When I use "streaming_query" to ask question @@ -163,15 +270,24 @@ Feature: Agent skills tests """ When I wait for the response to be completed Then The status code of the response is 200 - And The streamed response contains following fragments - | Fragments in LLM response | - | Unknown skill | #TODO: Make descriptive - + And The response is the last streamed fragment + And The body of the "tool_results" field is #TODO: Currently placeholder, should reflect actual tool results + """ + [ + { + "id": "1", + "status": "failure", + "content": "bla", + "type": "tool_result", + "round": 1, + } + ] + """ # --- Error handling: missing resource --- @SkillsConfig Scenario: load_skill_resource returns error for nonexistent resource file via query endpoint - Given The e2e-test-skill skill directory is present in the container + Given The e2e-test-skill skill directory path is "e2e-test-skill" And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration And The service is restarted When I use "query" to ask question @@ -179,11 +295,22 @@ Feature: Agent skills tests {"query": "Load references/nonexistent.md from e2e-test-skill using load_skill_resource.", "model": "{MODEL}", "provider": "{PROVIDER}"} """ Then The status code of the response is 200 - And The body of the response contains Resource not found + And The body of the "tool_results" field is #TODO: Currently placeholder, should reflect actual tool results + """ + [ + { + "id": "1", + "status": "failure", + "content": "bla", + "type": "tool_result", + "round": 1, + } + ] + """ @SkillsConfig Scenario: load_skill_resource returns error for nonexistent resource file via streaming_query endpoint - Given The e2e-test-skill skill directory is present in the container + Given The e2e-test-skill skill directory path is "e2e-test-skill" And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration And The service is restarted When I use "streaming_query" to ask question @@ -192,9 +319,19 @@ Feature: Agent skills tests """ When I wait for the response to be completed Then The status code of the response is 200 - And The streamed response contains following fragments - | Fragments in LLM response | - | Resource not found | + And The response is the last streamed fragment + And The body of the "tool_results" field is #TODO: Currently placeholder, should reflect actual tool results + """ + [ + { + "id": "1", + "status": "failure", + "content": "bla", + "type": "tool_result", + "round": 1, + } + ] + """ # --- Context management: deduplication --- @@ -209,21 +346,45 @@ Feature: Agent skills tests {"query": "Activate e2e-test-skill using the activate_skill tool.", "model": "{MODEL}", "provider": "{PROVIDER}"} """ Then The status code of the response is 200 - And I store conversation details + And I store conversation details + And The body of the "tool_results" field is #TODO: Currently placeholder, should reflect actual tool results + """ + [ + { + "id": "1", + "status": "success", + "content": "bla", + "type": "tool_result", + "round": 1, + } + ] + """ + When I use "query" to ask question with same conversation_id """ {"query": "Activate e2e-test-skill again using the activate_skill tool.", "model": "{MODEL}", "provider": "{PROVIDER}"} """ Then The status code of the response is 200 - And The body of the response contains already loaded #FIX: This looks/feels wrong (make more descriptve) + And The body of the "tool_results" field is #TODO: Currently placeholder, should reflect actual tool results + """ + [ + { + "id": "1", + "status": "failure", + "content": "bla", + "type": "tool_result", + "round": 1, + } + ] + """ # --- Multiple skills --- @SkillsMultiConfig Scenario: Skills directory path discovers all skills in subdirectories via query endpoint - Given The e2e-test-skill skill directory is present in the container - And The e2e-second-skill skill directory is present in the container + Given The e2e-test-skill skill directory path is "skills/e2e-test-skill" + And The e2e-second-skill skill directory path is "skills/e2e-second-skill" And The service uses the lightspeed-stack-skills-directory.yaml configuration And The service is restarted When I use "query" to ask question @@ -231,13 +392,23 @@ Feature: Agent skills tests {"query": "List all available skills using the list_skills tool.", "model": "{MODEL}", "provider": "{PROVIDER}"} """ Then The status code of the response is 200 - And The body of the response contains e2e-test-skill - And The body of the response contains e2e-second-skill + And The body of the "tool_results" field is #TODO: Currently placeholder, should reflect actual tool results + """ + [ + { + "id": "1", + "status": "success", + "content": "bla", + "type": "tool_result", + "round": 1, + } + ] + """ @SkillsMultiConfig Scenario: Skills directory path discovers all skills in subdirectories via streaming_query endpoint Given The e2e-test-skill skill directory path is "skills/e2e-test-skill" - And The e2e-second-skill skill directory is present in the container + And The e2e-second-skill skill directory path is "skills/e2e-second-skill" And The service uses the lightspeed-stack-skills-directory.yaml configuration And The service is restarted When I use "streaming_query" to ask question @@ -246,7 +417,16 @@ Feature: Agent skills tests """ When I wait for the response to be completed Then The status code of the response is 200 - And The streamed response contains following fragments - | Fragments in LLM response | - | e2e-test-skill | - | e2e-second-skill | \ No newline at end of file + And The response is the last streamed fragment + And The body of the "tool_results" field is #TODO: Currently placeholder, should reflect actual tool results + """ + [ + { + "id": "1", + "status": "success", + "content": "bla", + "type": "tool_result", + "round": 1, + } + ] + """