October 12

extremely stupid backup script (files+dbs)

#!/bin/bash

if [[ "$1" != "db_backup" && "$1" != "files_backup" && "$1" != "all" ]]; then
echo -e "Please run the script with following keys:\ndb_backup -> to do backup of all DBs\nfiles_backup -> to do backup of all users without DBs\nall -> to do users and their DBs backups at once"
exit 1
fi
datetime=$(date +%d-%m-%Y_%H-%M-%S)
db_backup_function(){
mkdir -pv /backup/dbs/"$datetime"
for i in `mysql -e "show databases;"|grep -wv "Database\|performance_schema\|information_schema\|sys\|mysql"`;do
mysqldump "$i" >> /backup/dbs/"$datetime"/"$i".sql
if [[ $? -ne 0 ]];then
echo "$datetime DB $i FAILED" >> /backup/failed_DBs.log
fi

zip --quiet /backup/dbs/"$datetime"/"$i".sql.zip /backup/dbs/"$datetime"/"$i".sql
if [[ $? -eq 0 ]];then
rm -f /backup/dbs/"$datetime"/"$i".sql
fi
done
}

files_backup_function(){
mkdir -pv /backup/files/"$datetime"
cd /home/
for i in `ls -1 /home/`;do
zip -r --quiet --symlinks /backup/files/"$datetime"/"$i".zip "$i"
if [[ $? -ne 0 ]];then
echo "$datetime files for $i account FAILED" >> /backup/failed_users.log
fi
done
}

if [[ "$1" = "db_backup" ]];then
db_backup_function
fi

if [[ "$1" = "files_backup" ]];then
files_backup_function
fi

if [[ "$1" = "all" ]];then
db_backup_function
files_backup_function
fi

October 12

Small reminder about commands

A few thread file transfer:

rclone copy :sftp:/disk_folder/disk_hash_id /the_way/to_target_folder/ --sftp-host=target_host_name_or_ip --sftp-user=your_username --sftp-ask-password --sftp-key-use-agent=false --sftp-shell-type=unix --sftp-md5sum-command=md5sum --progress --transfers=1 --multi-thread-streams=6 --checksum --bwlimit=300M

The single thread file transfer with additional validation:
rsync --progress --checksum --checksum-choice=md5 /disk_folder/disk_hash_id your_username@target_host_name_or_ip:/the_way/to_target_folder/

Aggregated itop stat (very helpful for a debug in some cases):

iotop -aoP

Manual checksums for the file:

md5sum /path/to/file

Look deeper at TCP timeouts and waits:

netstat -napo|grep TIME_WAIT

Check disk info:
qemu-img info disk_name

Create disk from command line:
qemu-img create -f qcow2 new_disk_name 15G

Converting/cloning the disk (It should also fix the dependency on the backing file, which means building a standalone VM)
qemu-img convert -O qcow2 old_with_backing_dependency.qcow2 new_standalone.qcow2

Add the backuing file to already existing disk (or rewrite this file):
qemu-img rebase -u -b /way_to_the_backing/file -F qcow2 /way_to_disk/new.qcow2

Creating disk WITH backing file:
qemu-img create -f qcow2 -F qcow2 -b /way_to_the_backing/file new.qcow2 500G

Resize disk:
qemu-img resize disk.qcow2 +10G

Attach the disk to VM:
virsh attach-disk --domain short_VM_ID /way_to_disk/file --target vdx --driver "qemu" --subdriver "qcow2"

Detach the disk from VM:
virsh detach-disk --domain short_VM_ID --target vdx

Check VM details:
virsh dumpxml short_VM_ID

Show all VMs:
virsh list --all