Optimize scraping

[luttje]

Currently we scrape the entire wiki when there's been changes. However it's perfectly achievable to only scrape the changed pages. We could then merge those pages with content of an existing release. This would save GitHub and the Facepunch wiki valuable resources.

I'm making note of this here, but placing it as low-priority for now.

[luttje]

wiki.facepunch.com/gmod/~pagelist?format=json's updateCount could work. I haven't checked if they update, but I would assume so.
It would be as simple as saving the count when scraping and comparing the count next time.

#31 (comment)

[sarahsturgeon]

Hello from https://github.com/CFC-Servers/gmodwiki! 👋
Big fan of yer snippets 😺

Gmodwiki is an optimized and self-hostable mirror for the official facepunch wiki.

We parse the changelist N (currently 4) times per day so that we can keep the mirror's content up-to-date.

A side effect of our build process is producing a single .zip that contains the .json of every page on the wiki. Currently, that file sits at ~20mb: https://storage.gmodwiki.com/public_cache.zip

We use this to speed up future builds, but perhaps it could be useful to snippets? You could skip the wiki scraping step and, instead, directly parse all of the .json files in our build artifact.

I'd be happy to make any changes to accommodate the snippets project.
We've been running gmodwiki for a little over a year, now and it's proven to be consistent and reliable 🙌

I'm sure this isn't high up on your priority list, but I happened across this Issue and figured I'd share.

Cheers! 🍻

[luttje]

Hey there,

Thanks, I really appreciate the offer!

After giving it some thought, I think it's best not to be too dependent on another service like Gmodwiki for the scraping results. It's not a question of trust in your service's reliability, but in the unlikely event it goes down, I'd prefer to avoid a situation where we're forced to revert to then outdated code just to get our own scraper back up and running.

One possible path forward could be to extract the scraping logic into a shared (npm) package that both projects can use. Or storing the json output in that separate repo, since we already rely on GitHub. That said, some of my other projects need attention right now, so I'm not sure I'd be able to contribute meaningfully in any other project. I also wouldn't want to freeload, so it's probably best to pin that idea for the time being.

Again, I truly appreciate the offer, and I'll give it a bit more thought in case I change my mind about using your JSON cache.

Thanks!

[sarahsturgeon]

Truthfully, I'd probably make the same call for the same reasons!

For awhile now, I've seen this fuzzy picture of some kind of overlap between the two projects. Every time I try to conceptualize something useful, I run into some sort of shortcoming or flaw.

For example, I once thought about running glua-api-snippets during Gmodwiki's build and uploading the library/ somewhere.
My original goal was to give folks a way to keep their local snippets updated with fresher types than is reasonable for the snippets project to provide. Unfortunately I can't think of any good way to actually get that artifact to users - especially for folks using the vscode extension.

Your idea of using github as a medium is wise, and I could see potential there. With some fairly hefty modifications on both projects it could work, but the cost:reward ratio isn't quite tipped in our favor there, either.

Alas. Maybe we'll find a clearer picture of this idea someday :)

Thank you for entertaining the thought!