Up until recently I have been backing up my most important data to Backblaze. I wanted to implement something that was more of a last resort, cold storage backups to backup both my servers and my file share. I wanted this to be easy to implement, and easy to restore in the event of a total loss of data. I manually do this every 3 months, alternating between sets of disks. I will demonstrate how I implement this. You will need a USB hard drive.
First we need get the hard drive to which we will backup, and attach it via USB adapter. The disk will obviously need to be large enough to fit all the VM backups. Plug it into the first node to backup. In the Proxmox web UI, go to the node that the drive is attached to and navigate to the node > Disks. Go ahead and wipe and then initialize the disk you just plugged in. You can add the drive either through the UI by going to Disks and then wiping the disk you connected, or you can go to Shell and use cfdisk then use the mk2fs.ext4 command. Now that the disk is prepared we want to mount it and add it to Proxmox.
Go to Shell in each node and create the mount points for the drive, then mount it on the node it's attached to. Make sure you have the correct device:
mkdir /mnt/Cold-Backups
mount -t ext4 /dev/sdf1 /mnt/Cold-Backups
You could do this on the first node by going to Disks > Directory > Create: Directory, but I think you'll still need to manually create each mount point on the other nodes.
To make this storage usage, use this command to add the storage to be able to use in the cluster:
pvesm add dir Cold-Backups --path /mnt/Cold-Backups --content backup
Now you should be able to go to each VM in the node that the drive is connected, go to the backup section, and backup directly to the disk. Make sure that what you are backing up to is the disk you have mounted - if the disk is not mounted to /mnt/Cold-Backups or you are on the wrong node, you will be backing up to the host's system drive by accident. Check this by going to the node then checking under the Cold-Backups storage to see if it has the correct space listed.
When all the VMs are backed up from this node unmount the disk and repeat the process for the remaining nodes: umount /dev/sdf1
You can remove the storage from the cluster now if you want. When you want to back up again you can run the pvesm command from earlier and mount the disk back to /mnt/Cold-Storage. I just leave everything as is, that way I just need to plug the drive back in, nice and easy.
I ended up doing something similar in my Proxmox Backup Server. The setup is very similar: I prepared the disk in Shell using cfdisk to wipe the disk and create a fresh new partition, the mkfs.ext4 to crate the filesystem. I created the mount point and mounted the drive. Then I went to add datastore - I removed the datastore after I was finished with everything to avoid the OS getting hung up on trying to find it while booting. From there it was just a matter of adding it in Proxmox.