Add a system for regular and/or on-demand data dumps

[Oaphi]

meta:295094
meta:293597

Adding a tracking issue since it's been requested a couple of times and is a general expectation of our community members. It won't be an easy feat to implement properly, but we shouldn't lose sight of the request.

Related: meta:285446

[ArtOfCode-]

This is actually not as difficult as it appears at first. I've done both data dumping and live SQL data explorer elsewhere, effectively by whitelisting specific columns and using views to limit access.

[Oaphi]

Oh, that's nice! I haven't inspected Metasmoke's code in a while - this quite simplifies things

[Oaphi]

I suppose the main thing now is to decide what to include in the dump. There are some access control considerations - I presume we can only include public categories & related entities.

[cellio]

Related: #780

[SuperCowProducts]

I suppose the main thing now is to decide what to include in the dump. There are some access control considerations - I presume we can only include public categories & related entities.

My (duplicate) post focused on using it as a learning / reference resource, so for that use case the key items would be questions, answers and votes. It would be interesting to hear others' needs!

[cellio]

For that purpose, we'd probably also want to capture reactions (e.g. outdated) and comments.

[SuperCowProducts]

Is there anything users interested in this can help with? I think if any kind of data that is already public on the website gets released, people will eventually figure out a good way to extrapolate what they need. For reference, I've been doing something similar on my own repo, and would love to build something that reconstructs the Q&As in a human-readable format from raw data.

This is actually the feature Codidact will win me over, which is why I'm trying to make it a reality.