Add native windows launcher to replace the current .bat files

[sbc100]

Its not clear if this will lead to issues with virus/security scanners. For this reason, I'm leaving the old .bat files
around for now and user can opt out of this by running tools/maint/create_entry_points.py --bat-files.

Assuming we don't run into any issues I hope to eventually completely remove support for the .bat files.

There are several reasons to want to do this:

1. Batch files are notoriously painful to work with and mis-understood.
2. Should be faster (no need to launch cmd.exe).
3. Works around several known issues with .bat files including one that is known unsolvable one (see emcc.bat for more details)
4. In the future we could potentially launch clang directly, bypassing the python completely (or at least only invoking python for the link stage).

See: #26229
See: #19207

[sbc100]

@dschuff @kripken WDYT?

[kripken]

I don't know much about windows, but would we need to build this as part of the emsdk, ship it, etc.? How important are the issues solved by this?

[sbc100]

I don't know much about windows, but would we need to build this as part of the emsdk, ship it, etc.? How important are the issues solved by this?

I think since its just a tiny .exe we could just check it into git once.. then the create_entry_points.py script would make N copies of it, one for each script.

[kripken]

I think since its just a tiny .exe we could just check it into git once..

Isn't there windows-on-arm, so a single binary wouldn't work?

[sbc100]

I think since its just a tiny .exe we could just check it into git once..

Isn't there windows-on-arm, so a single binary wouldn't work?

True! Although I'm not sure anyone is using that yet. We certainly don't yet provide emsdk binaries for windows/arm64.

[dschuff]

I haven't reviewed the code yet but I think this idea is generally good We did something similar for the NaCl compiler (for a different reason, but a similar technique). I think I have a mild preference for building it with emsdk rather than checking it in, but I could be convinced.

[dschuff]

Does this work with non-ascii directory/path names?

[cwoffenden]

There's a whole world of TCHAR pain and Win32 file API coming someone's way...

[sbc100]

The new versions handles all that I believe.

[RReverser]

3. Works around several known issues with .bat files including one that is known unsolvable one (see emcc.bat for more details)

FWIW PowerShell scripts I added a while back should also already solve this issue, and they're chosen by default on Windows when users type emcc ... in PowerShell.

I don't mind the .exe approach, it might make it easier to solve the long-standing issue with Ctrl+C as well - that one is somewhat difficult to solve with either of scripting systems - just want to make sure it's not a burden to maintain.

[RReverser]

We certainly don't yet provide emsdk binaries for windows/arm64.

Windows have long had system similar to macOS Rosetta, it's just that Windows on ARM has only picked up in popularity recently with new laptops, but essentially any x64 .exe is automatically supported on ARM via translation layer.

Native ARM might be nice to have for wasm-binaries, but for something as simple as this invocation script perf shouldn't matter as much.

[sbc100]

3. Works around several known issues with .bat files including one that is known unsolvable one (see emcc.bat for more details)

FWIW PowerShell scripts I added a while back should also already solve this issue, and they're chosen by default on Windows when users type emcc ... in PowerShell.

I don't mind the .exe approach, it might make it easier to solve the long-standing issue with Ctrl+C as well - that one is somewhat difficult to solve with either of scripting systems - just want to make sure it's not a burden to maintain.

Having to maintain the .ps1 and .bat files in parallel is another reason to make this switch.

Sadly I don't think the .ps1 files can actually replace the .bat files because IIRC powershell is somehow optional in windows so its not guaranteed to always be there.

[RReverser]

Perhaps worth looking at py2exe than rolling something own?

[sbc100]

Perhaps worth looking at py2exe than rolling something own?

I have looked at things like that over the years but the complexity always scared me off. It would also be little sad to loose the "edit-in-place" abilities of the plain-old-python files.

[sbc100]

I asked Gemini CLI to help me write a better version. Its seems to have done it!

I asked it to find any discrpencies between these two and it came
up with:

launcher.c Analysis:

The C code replaces the batch script by directly launching Python or ccache. It correctly determines the Python executable and the script to run. Key features include:

• Long Path Support: It uses the \?\ prefix, removing the 260-character MAX_PATH limit, a major improvement over the batch script.
• Robust Path Discovery: It uses GetModuleFileNameW to reliably find its own path, avoiding the complex workarounds present in the batch script for PATH lookups.
• ccache Handling: The ccache logic is similar to the batch script.
• Argument and Stdin Handling: It uses the robust CommandLineToArgvW for arguments and a more reliable method for the Python stdin bug workaround.

Potential Issues & Discrepancies:

1. Filename Length Limits: The C launcher is superior. It removes the MAX_PATH limitation of the batch script, so there are no new length restrictions. This is a clear advantage.

2. Features Missing from launcher.c:

   • EM_WORKAROUND_WIN7_BAD_ERRORLEVEL_BUG: The batch script contains a workaround for a Windows 7 bug causing incorrect exit codes. The C launcher, despite being compiled for Windows 7
     compatibility, lacks this workaround. This is a potential regression for Windows 7 users.
   • %~f0 / %~dp0 Workaround: The batch script's workaround for a PATH lookup bug is obsolete, as the C launcher's method of finding its path is more reliable. This is an improvement.

Conclusion:

The primary concern is the missing EM_WORKAROUND_WIN7_BAD_ERRORLEVEL_BUG handling in launcher.c. This could cause issues for users on Windows 7. Otherwise, the C launcher is a significant
improvement, particularly in its handling of long file paths and command-line arguments. It does not introduce any new filename length limitations. I will now prepare my final answer based on
this.

[sbc100]

Perf number seem fairly negligable:

.exe:

[1/1] test_emcc_noop (test_benchmark.benchmark.test_emcc_noop) ... 

        clang: mean: 0.033 (+-0.008) secs  median: 0.037  range: 0.022-0.042  (noise: 24.282%)  (5 runs)   Relative: No baseli
ne recorded yet
         emcc: mean: 0.259 (+-0.008) secs  median: 0.255  range: 0.251-0.270  (noise: 3.041%)  (5 runs)   Relative: 7.73 X slo
wer
ok

.bat:

[1/1] test_emcc_noop (test_benchmark.benchmark.test_emcc_noop) ... 

        clang: mean: 0.035 (+-0.007) secs  median: 0.037  range: 0.024-0.042  (noise: 18.888%)  (5 runs)   Relative: No baseli
ne recorded yet
         emcc: mean: 0.287 (+-0.011) secs  median: 0.292  range: 0.265-0.294  (noise: 3.952%)  (5 runs)   Relative: 8.17 X slo
wer
ok

[sbc100]

@dschuff @kripken @juj, I'd like to land this today and get it running out our bots and in the hands of users.

I will send out a message to mailing list.

If we run into the security scanner issues we can investigate more or revert.

[dschuff]

I wonder if we should build the exes with the same compiler we use to build the emsdk binaries, just to minimize the chances of there being any interesting differences. I'm also not super enthusiastic about checking in binaries, but I guess if we want to maintain the invariant that you can just check out emscripten and use it without emsdk or other dependencies, then there aren't too many other options?

[sbc100]

I wonder if we should build the exes with the same compiler we use to build the emsdk binaries

Yes, perhaps we can do that on the emscripten-releases waterfall? I think that can happen later though, if we do run into issue.

I'm also not super enthusiastic about checking in binaries, but I guess if we want to maintain the invariant that you can just check out emscripten and use it without emsdk or other dependencies, then there aren't too many other options?

Yes, I'm normally completely against that too, but I think this is good example of case where an exception that rule makes sense. Since the binary is so small we can think of it more like a binary shell script :)

[dschuff]

Since this is a fairly significant change, I wonder if there is a way we can make it "opt-in" first. Could we have both exe and bat files for a couple of releases and have people test?

[sbc100]

Since this is a fairly significant change, I wonder if there is a way we can make it "opt-in" first. Could we have both exe and bat files for a couple of releases and have people test?

I did keep the ability to opt out, but having them side-by-side complicates things I think.

Are you suggesting something like:

1. Install both .bat and .exe launchers
2. Have our internal systems looks for one overt the other, with the order dependent on some setting (environment variable?)?

Note: I don't think that this really works since users who just type emcc will be at the whim of the OS as to which one is selected.

I guess you don't like the current opt out mechanism of having to re-run the create_entry_points.py script?

I guess one major downside of that mechanism is that the script does not exist in installed versions of emscripten (such as emsdk installs) because the maint directory is not included?

[dschuff]

Since this is a fairly significant change, I wonder if there is a way we can make it "opt-in" first. Could we have both exe and bat files for a couple of releases and have people test?

I did keep the ability to opt out, but having them side-by-side complicates things I think.

Are you suggesting something like:

1. Install both .bat and .exe launchers
2. Have our internal systems looks for one overt the other, with the order dependent on some setting (environment variable?)?

Yeah I guess so.

Note: I don't think that this really works since users who just type emcc will be at the whim of the OS as to which one is selected.

Speaking of which, which one would actually be selected? Seems like it ought to be deterministic.
But this use case is probably rare, I would imagine CMake and more direct use are much more common.

I guess you don't like the current opt out mechanism of having to re-run the create_entry_points.py script?

It's not bad. Just wondering if it could be eaiser.

I guess one major downside of that mechanism is that the script does not exist in installed versions of emscripten (such as emsdk installs) because the maint directory is not included?

Yeah, a version that didn't require e.g. repackaging the emsdk install after running the script would be best. but yeah it seems like the opt-out should work for emsdk users if possible since that's most users.

[sbc100]

I think I'd like to land this now while we are early in the 5.0.3 cycle. If we run into any teething issues we will have time to either address them or improve the fallback so that normal emsdk users can use it too.

I think trying to figure out how to include the fallback in the shipped version is more work that needs to be done at this point.

[juj]

Great to see work on this Sam. Though I do have to say I expect this to be a source of regression trouble.

Fixes: #26229

My gut feeling here is that the .exe files will have the same long path limitation as the .bat files. I.e. the long path limitation on Windows is a limitation of the terminal, and not a limitation of running .bat files. Though I could be wrong.

Fixes: #19207

Iiuc this issue was resolved already by the earlier PR?

In any case, I have been working towards code signing the compiler executables at Unity, so (presuming I get that done before the next time we have to update) the Windows SmartScreen issue will likely not be a trouble for us at least, if I can manage to code sign the .exe wrappers for our distribution.

LGTM overall. Good to see there is a fallback in interim.

I was hoping a main benefit of this would be performance, as there would be some CPU overhead on Windows from having to launch the .bat files.. but reading the comment thread, that perf improvement apparently did not manifest? That is a bit disappointing.

So I am not sure if I see a good benefit for this, however, happy to see this land, if nothing else, to learn more how well this kind of approach will work (and to understand if using .exe instead of .bat will indeed avoid having to use .rsp files)

[juj]

My own local test lab CI is a bit heavily on the red atm.. I'll try to see if I can get it to good state again, so it can test out this on Windows on ARM as well.

[sbc100]

My gut feeling here is that the .exe files will have the same long path limitation as the .bat files. I.e. the long path limitation on Windows is a limitation of the terminal, and not a limitation of running .bat files. Though I could be wrong.

Apparently .bat files do have a lower limit on command line length (8k chars I think) whereas windows has a much larger limit, but still limited compared to UNIX. This was pointed out to me in one of the comments above. So yes, I suppose it doesn't actually fully resolve that issue, but it will increase out limit to the same as other compilers on windows (e.g. gcc, clang, cl.exe).

[sbc100]

Fixes: #19207

Iiuc this issue was resolved already by the earlier PR?

True, but that PR (#19212) sill contains a fatal error in some cases (echo Fatal Error! Due to a Windows bug, we are unable to locate the path to %~n0.bat.) so I assumed it was still conceivable that someone might hit that?

[sbc100]

Fixes: #19207

Iiuc this issue was resolved already by the earlier PR?

True, but that PR (#19212) sill contains a fatal error in some cases (echo Fatal Error! Due to a Windows bug, we are unable to locate the path to %~n0.bat.) so I assumed it was still conceivable that someone might hit that?

Also, just removing that workaround code is nice to have.

[sbc100]

Regarding the perf gains, I'm still now sure if we will see gains. I'll be curious to see, for example, if that full test suite speeds up on out waterfall (and your yours, @juj)