Copy the store to GCS¶
The store keeps one current state on the cluster and a full copy in a Google Cloud Storage
(GCS) bucket. Any full copy is the store, so work can go on from the bucket if the cluster
copy is lost. The tool that moves files is rclone, which comes with the pixi environment.
Set up once¶
Make an rclone remote of type Google Cloud Storage, here called gcs:
Then name the place in the bucket in the store's labdata-store.toml:
Log in each day¶
Your Google login for rclone lasts one day. Renew it before a copy:
A cluster node has no web browser. Answer n when rclone asks to use one. It then prints an
rclone authorize ... command. Run that command on a computer with a browser, log in, and
paste the token it prints back into the prompt on the node.
Prune and copy¶
prune deletes builds and snapshots the current build no longer reads. It keeps each one for
a grace period after it goes out of use, so a reader can finish; the
API reference gives its length. --grace 0 deletes
them at once: use it only as a one-off to tidy a store nobody is reading.
copy sends the marker file, the store's README.md, the current snapshots, raw/ and the
current build, and deletes anything else in the bucket's copy. It leaves out work/,
support/ and refresh-status.json. It will not run unless the current build is complete.
It first checks the login with one listing call. When the login has expired, it stops and
exits cleanly:
rclone's Google login has expired, so the bucket is behind the store. Run `rclone config reconnect gcs:`, then `labdata store copy`.
Log in, then run labdata store copy again. Files already sent are skipped. A refresh that
meets an expired login still succeeds, and marks the copy pending in refresh-status.json
until labdata store copy succeeds.
Check the copy¶
Compare the store with its copy_to, leaving out what the copy leaves out:
pixi run rclone check "$LABDATA_STORE" <copy_to> --one-way \
--filter "- .*" --filter "- .*/**" --filter "- /work/**" \
--filter "- /support/**" --filter "- /refresh-status.json" \
--filter "- /builds/<old build>/**"
It should report 0 differences. A build or snapshot that prune keeps for its grace period is not in the bucket: add a filter for each one instead of counting it as a difference.
Read the store on another computer¶
Copy the store down, check it against the bucket, and point labdata at it:
rclone copy gcs:<bucket>/<store> /data/store
rclone check gcs:<bucket>/<store> /data/store
export LABDATA_STORE=/data/store
To only read the store, copy just the marker and the build it names: