Update build scripts for WebAssembly version of gap-system for faster startup and on-demand loading

[wangyenshu]

Summary

Update build scripts for WebAssembly version of gap-system for faster startup and on-demand loading. If files are in startup_manifest.json, it is preloaded into IDBFS; otherwise files are loaded on-demand into memory. By applying this strategy, the startup time of WebAssembly version of gap-system with "bootstrap-pkg-full" is about 10s on my laptop on both Firefox and Chrome. And it only downloads about 10MB resources on startup.
Try this "bootstrap-pkg-full" example here:
https://wangyenshu.github.io/gap-wasm/
Web files:
https://github.com/wangyenshu/gap-wasm/tree/gh-pages

Text for release notes

none

Further details

All resource path ("pkg lib grp tst doc hpcgap dev benchmark") are hashed because there are files with special characters, e.g., "pkg/simpcomp/complexes/manifolds/3Manifolds/(S2twistS1)#2#RP3.scb". And GitHub pages does not allow that.

The startup_manifest.json is generated using the build_startup_manifest.js script.

The scripts for hashing are
generate_mapping.py
copy_hashed_assets.py
These scripts are not perfect, and they will break if filenames contain backslash or whitespace. Fortunately, there are no such files in these directories:"pkg lib grp tst doc hpcgap dev benchmark".

I think this can serve as a good online demo on the gap-system website.

[fingolfin]

Wow, that's pretty amazing!

[fingolfin]

Do we really need to commit this script? Can't it be generated as part of the build step (and put into .gitignore)

[wangyenshu]

It is up to you. The file startup_manifest.json can be generated using the build_startup_manifest.js script but that is not automated. This script should be placed in the web root dir and will open a server on port 9999 and will record all file path that is requested to the server. The user need to manually visit 127.0.0.1:9999, wait for the start-up stage to finish, then terminate the script manually.

If there will be some signals when GAP finished loading, it will be possible to automate it using tools like Playwright.

[ChrisJefferson]

Just to say, this is really cool, I'm very glad you got it to work correctly, I could never get dynamic loading to function.

[fingolfin]

Just some thoughts (which are not meant to take away an iota of the wonderful achievement of you and @ChrisJefferson in creating this):

For a website demo, this needs a little bit more "chrome" or polish: e.g. there should be some text on the page that indicates what this is, possibly with a link back to the GAP website.

Of course if it is on a page on www.gap-system.org itself, one would expect the usual styling of that website to be applied (that should be doable). I.e. that might not be too much work.

Maybe the text could also include a few usage hints. E.g. one has to click into the "terminal" to be able to type stuff. The text could also suggest something to try out, and perhaps mention some limitation ("All GAP packages are available that do not rely on compiled code, or other GAP packages using compiled code" or so)

[fingolfin]

Another thing: is it really necessary to hash all filenames? HTTP has another way of dealing with filenames that contain special characters: percent encoding (e.g. %20 for a space. The advantage would be that most filenames would actually stay unchanged.

[fingolfin]

Great, even the help system works (tried ?Group). Although it took about 30 seconds for me. The "Network" view of theWeb Inspector of my browser revealed the issue: the help system files are not prefetched, and the help system tries to greedily load a gazillion .six files (well... 177: one per package plus some for GAP manuals), one by one, which is slooow due latency. I.e.: 177 small requests is slower than 1 big request. Or even 177 requests that are made concurrently.

Of course then my browser had those files in the cache, so for a reload, the overhead per file was much slower, but it was still noticable

I wonder if it would make sense to also prefetch all manual.six files?

[wangyenshu]

Just some thoughts (which are not meant to take away an iota of the wonderful achievement of you and @ChrisJefferson in creating this):

For a website demo, this needs a little bit more "chrome" or polish: e.g. there should be some text on the page that indicates what this is, possibly with a link back to the GAP website.

Of course if it is on a page on www.gap-system.org itself, one would expect the usual styling of that website to be applied (that should be doable). I.e. that might not be too much work.

Maybe the text could also include a few usage hints. E.g. one has to click into the "terminal" to be able to type stuff. The text could also suggest something to try out, and perhaps mention some limitation ("All GAP packages are available that do not rely on compiled code, or other GAP packages using compiled code" or so)

Exactly. I think the simplest way to do that is to use iframe. For example, host this GAP-wasm in another repo and use iframe to embed only the canvas (terminal) of GAP-wasm into some proper place of www.gap-system.org. In that case, maybe some hacks should be done to make COOP and COEP works.

Another thing: is it really necessary to hash all filenames? HTTP has another way of dealing with filenames that contain special characters: percent encoding (e.g. %20 for a space. The advantage would be that most filenames would actually stay unchanged.

That is right. Maybe I can improve it in the future.

Great, even the help system works (tried ?Group). Although it took about 30 seconds for me. The "Network" view of theWeb Inspector of my browser revealed the issue: the help system files are not prefetched, and the help system tries to greedily load a gazillion .six files (well... 177: one per package plus some for GAP manuals), one by one, which is slooow due latency. I.e.: 177 small requests is slower than 1 big request. Or even 177 requests that are made concurrently.

Of course then my browser had those files in the cache, so for a reload, the overhead per file was much slower, but it was still noticable

I wonder if it would make sense to also prefetch all manual.six files?

Yes, that sounds good. I can modify build_startup_manifest.js to include all files whose suffix is ".six".

[wangyenshu]

I convert this pull request to draft. I will re-open it when I finish implementing the above improvements.

[wangyenshu]

Update:
Now resource directory and filename is encoded with percentage encoding. For example, if the original path is
pkg/jupyterviz/examples/EV Charge Points.json, then the file stored on github pages is
pkg/jupyterviz/examples/EV%20Charge%20Points.json and the url for requesting that file is
pkg/jupyterviz/examples/EV%2520Charge%2520Points.json.

I have updated the above changes to https://wangyenshu.github.io/gap-wasm/. And it also preloads 170/173 manual files. The three missing files are listed here:

assets/pkg/hap/tutorial/manual.six
assets/pkg/gapdoc/3k%2B1/manual.six (this is /pkg/gapdoc/3k+1/manual.six before encoding)
assets/tst/mockpkg/doc/manual.six

The new startup_manifest.json is built in the following ways:

• put the build_startup_manifest.js in web root and run it
• open localhost:9999 and wait for gap to initialize
• run the following command in the gap terminal to load most of manual files:

SizeScreen([100000, 100000]);
??a

I am not sure if these are the best approaches. Any feedback is greatly appreciated.

---

The following code snippet can be used (or serve as a reference) for embedding WebAssembly version of GAP on website:

<script>
  document.getElementById('btn-launch-gap').addEventListener('click', function() {
    const container = document.getElementById('gap-container');
    
    const iframe = document.createElement('iframe');
    iframe.src = "HOST_URL"; 
    
    iframe.style.width = "100%";
    iframe.style.height = "500px"; 
    iframe.style.border = "none";
    iframe.style.backgroundColor = "#000"; 
    iframe.style.display = "block";
    
    container.innerHTML = '';
    container.appendChild(iframe);
  });
</script>
<script src="coi-serviceworker.js"></script>

where HOST_URL should be replaced by the URL of WebAssembly version of GAP.