Optimize sync performance by skipping npm install when package.json is unchanged

[jessejackson85]

Problem

Currently, the /sync endpoint runs npm install --force every time it processes a JavaScript/TypeScript repository, even when there are no changes to package.json. This causes unnecessary processing time and resource usage, especially for frequently synced repositories.

Solution

This PR optimizes the sync process by:

1. Comparing package.json content before and after cloning the repository
2. Skipping the npm install command when package.json hasn't changed
3. Using environment variables to control this behavior
4. Adding a dedicated helper function for sync operations

Implementation Details

• Added direct file comparison in clone_repo() to detect package.json changes
• Added conditional logic in run_cmd() to skip npm install commands when appropriate
• Created get_use_lsp_for_sync() utility function to properly set all environment variables
• Updated process(), sync_async() and related functions to use the new optimization
• Added logging for better visibility and debugging
• Ensured regular ingests continue to perform full installs

Testing

Tested with multiple scenarios:

• Initial sync (always installs packages)
• Repeated sync without package.json changes (skips npm install)
• Sync after modifying package.json (performs npm install)
• Regular ingest operations (always installs packages)
• Async endpoints

Closed: #390

Expected Benefits

• Performance: Significantly reduces sync time for repositories with unchanged dependencies
• Resource Usage: Lowers CPU and memory consumption for sync operations
• Responsiveness: Faster response times for users syncing repositories
• Consistency: Maintains correct behavior when dependencies actually change

Screenshots/Logs

Before optimization:

Cloning repo to "/tmp/stakwork/sphinx-tribes"
Running cmd: "npm install --force"
... [long npm install output] ...
==>> Total processing time: 37.23s

After optimization (repeated sync):

Cloning repo to "/tmp/stakwork/sphinx-tribes"
=> package.json unchanged, skipping npm install
==>> Total processing time: 5.17s