I replaced both of the drives in my RAID1 boot array on a Ubuntu server yesterday and had some struggles so I thought I would write down my experience both for my future self and any others who needs to do something similar. I think my main problem was following a guide that was not for UEFI setups \ud83d\ude42<\/p>\n\n\n\n
So, my setup is two NVMe drives on nvme0n1 and nvme1n1. Both drives have the same partition table:<\/p>\n\n\n\n
Before starting, check that all disks are online:<\/p>\n\n\n\n
# cat \/proc\/mdstat\n Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] \n md1 : active raid1 nvme1n1p3[2] nvme0n1p3[3]\n x blocks super 1.2 [2\/2] [UU<\/strong>]\n bitmap: 3\/7 pages [12KB], xKB chunk\n md0 : active raid1 nvme0n1p2[3] nvme1n1p2[2]\n x blocks super 1.2 [2\/2] [UU<\/strong>]<\/pre>\n\n\n\nSo it should say [UU] and not [U_] or something else.<\/p>\n\n\n\n
In my case I also did
smartctl -ia \/dev\/nvme0<\/code> to write down serial numbers etc to know which physical disk to replace in the box.<\/p>\n\n\n\nNext up is to save a copy of your EFI partition for later use:<\/p>\n\n\n\n
dd if=\/dev\/nvme0n1p1 bs=4M of=\/root\/EFI_PARTITITON.img<\/pre>\n\n\n\nPick one drive to start with and make it offline:<\/p>\n\n\n\n
mdadm \/dev\/md0 --fail \/dev\/nvme0n1p2 --remove nvme0n1p2\nmdadm \/dev\/md1 --fail \/dev\/nvme0n1p3 --remove nvme0n1p3<\/pre>\n\n\n\nPhysically replace the drive in the server. It might also be a good idea to do a firmware upgrade of the new drive if it’s available.<\/p>\n\n\n\n
After disk is replaced, copy over the partition table from the remaining good disk to the newly replaced disk, and then generate new UUIDs for the partitions:<\/p>\n\n\n\n
sfdisk -d \/dev\/nvme1n1 | sfdisk \/dev\/nvme0n1\nsgdisk -G \/dev\/nvme0n1<\/code><\/pre>\n\n\n\nAdd the device back to the RAID1 array:<\/p>\n\n\n\n
mdadm --manage \/dev\/md0 --add \/dev\/nvme0n1p2\nmdadm --manage \/dev\/md1 --add \/dev\/nvme0n1p3<\/pre>\n\n\n\nMonitor the rebuild status with a command like
watch cat \/proc\/mdstat<\/code><\/p>\n\n\n\nIf the rebuild is slow\/limited to like 200MB\/s you can change the max speed to eg 1.6GB\/s with:<\/p>\n\n\n\n
echo 1600000 > \/proc\/sys\/dev\/raid\/speed_limit_max<\/pre>\n\n\n\nAfter the resync is done we need to fix EFI, start by copying back the partiton we backed up before:<\/p>\n\n\n\n
dd if=\/root\/EFI_PARTITITON.img bs=4M of=\/dev\/nvme0n1p1<\/pre>\n\n\n\nFix grub:<\/p>\n\n\n\n
update-grub
grub-install \/dev\/nvme0n1<\/pre>\n\n\n\nAnd lastly reinstall the UEFI boot option (for ubuntu):<\/p>\n\n\n\n
efibootmgr -v | grep ubuntu # only shows one entry\nefibootmgr --create --disk \/dev\/nvme0n1 --part 1 --label \"ubuntu\" --loader \"\\EFI\\ubuntu\\shimx64.efi\"<\/pre>\n\n\n\nYou should after this have two ubuntu entries from efibootmgr, and that should be it! Try rebooting and make sure boot works from both drives. If you need to replace the other drive follow the same procedure as above but use eg nvme1n1.<\/p>\n\n\n\n
If the new drive is bigger you can also grow the RAID1:<\/p>\n\n\n\n
mdadm \/dev\/md1 --fail \/dev\/nvme0n1p3 --remove nvme0n1p3
parted \/dev\/nvme0n1<\/pre>\n\n\n\nUse
resizepart<\/code> command inparted<\/code> to change the size.<\/p>\n\n\n\nAdd back to the array:<\/p>\n\n\n\n
mdadm --manage \/dev\/md1 --add \/dev\/nvme0n1p3<\/pre>\n\n\n\nWait for resync (cat \/proc\/mdstat) and then grow the array:<\/p>\n\n\n\n
mdadm --grow \/dev\/md1 --size=max<\/pre>\n\n\n\nThen you need to resize the filesystem, in my case it’s LVM with ext filesystem on top so:<\/p>\n\n\n\n
pvresize \/dev\/md1
lvextend -L +1G \/dev\/vg-md1\/lv-root
resize2fs \/dev\/vg-md1\/lv-root<\/pre>\n\n\n\nOr just resize2fs \/dev\/md1 if it’s no LVM.<\/p>\n","protected":false},"excerpt":{"rendered":"
I replaced both of the drives in my RAID1 boot array on a Ubuntu server yesterday and had some struggles so I thought I would write down my experience both for my future self and any others who needs to … Continue reading