backups not possible when node is only

[frank-bee]

To be able to recover a full node fast and easy, if would be nice to allow full backups in an automatic way: Creating disc snapshots instead of application / database backups.

I tried this on AWS with automated backups , which creates backups while the vm is running.

Trying to relaunch the node from such a backup leads to the following errors

Error: unit not found: RWxLmESDr7........B1nFk/ljbKQ=
    at Object.ifNotFound (/home/obyte/obyte-hub/node_modules/ocore/storage.js:1685:13)
    at /home/obyte/obyte-hub/node_modules/ocore/storage.js:54:21
    at /home/obyte/obyte-hub/node_modules/ocore/kvstore.js:30:13
    at /home/obyte/obyte-hub/node_modules/levelup/lib/levelup.js:164:14
    at /home/obyte/obyte-hub/node_modules/encoding-down/index.js:51:21
(node:3467) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 ready listeners added. Use emitter.setMaxListeners() to increase limit
/home/obyte/obyte-hub/node_modules/ocore/storage.js:1685
                        throw Error("unit not found: "+row.unit);

Probably the obyte core is not able to deal with small database inconsistencies?

[tarmo888]

You can't have distributed ledger that has database inconsistencies. You need to stop the node, do the backup and start the node again.

[tonyofbyteball]

This is most likely the result of an inconsistent backup. Sqlite database consists of several files and if copies of these files are created at different time, the copy gets inconsistent.

There are some filesystems, such as ZFS, that allow taking snapshots. This would guarantee that the copy is consistent.

[frank-bee]

I would prefer to use an external database system which handles the backups transparently.
Would if be difficult to allow this? Ideally MySQL or Posgre
One advantage of using external database in obyte would be dockerization, should be straight forward if the rest of the application is stateless. we could even run obyte node in a k8s cluster

[tarmo888]

Isn't docker containers able to use external drives, for the data folder? You can have code and data separated.

You could backup SQLite with .backup command also when it is running and doesn't have much load on it. That would give a consistent backup of SQLite file, but that wouldn't solve anything either because that database would not be consistent with RocksDB data because by the time backup finishes, it has already written more data to it.

MySQL is possible to use, but that won't replace RocksDB and I don't think it will solve anything either. It probably make you more trouble because MySQL implementation is not as tested as SQLite one and you would need to do migrations manually. MySQL doesn't perform as good as SQLite even on beefed up machine, which would cost lot more. Probably because it is well optimized for SQLite.