Tracking issue for backup support in S3d. Right now s3d requires manually backing sqlite which is inconvenient and makes it annoying to use s3d as a backup source.
To fix this we can do the following
- Add an endpoint to create a sqlite backup (copy hostd behavior)
- Add an endpoint to flush objects that haven't been uploaded yet regardless of padding
- Add CLI cmds to flush files and create, list and delete backups
- Add a recovery command which syncs all objects, lists the snapshot ones, let's the user select one of them and then creates a database from it
- Update the README to explain how it all works and what the implications are
This is quite a bit of code and should be done in multiple PRs. e.g. creating the backup can be one, the flush endpoint can be one, the commands and finally the recovery command that restore the database.
NOTE: One thing we need to be careful about is making sure that orphaning doesn't break past backups. e.g. if we create a backup and then delete a file, restoring should result in the deleted file still being pinned. So we need to 1. not orphan an object if it is linked to a snapshot and 2. potentially insert orphans when we delete a snapshot
NOTE2: Restoring a snapshot that isn't the latest one will also require careful consideration since snapshots are not independent. e.g. restoring snapshot 1 leaves s3d without knowledge of snapshot 2 and all of its pinned objects. Meaning that in the worst case using snapshot 1 corrupts snapshot 2 and in the best case we collect garbage data.
Tracking issue for backup support in S3d. Right now s3d requires manually backing sqlite which is inconvenient and makes it annoying to use s3d as a backup source.
To fix this we can do the following
This is quite a bit of code and should be done in multiple PRs. e.g. creating the backup can be one, the flush endpoint can be one, the commands and finally the recovery command that restore the database.
NOTE: One thing we need to be careful about is making sure that orphaning doesn't break past backups. e.g. if we create a backup and then delete a file, restoring should result in the deleted file still being pinned. So we need to 1. not orphan an object if it is linked to a snapshot and 2. potentially insert orphans when we delete a snapshot
NOTE2: Restoring a snapshot that isn't the latest one will also require careful consideration since snapshots are not independent. e.g. restoring snapshot 1 leaves s3d without knowledge of snapshot 2 and all of its pinned objects. Meaning that in the worst case using snapshot 1 corrupts snapshot 2 and in the best case we collect garbage data.