From owner-linux-raid-outgoing@vger.rutgers.edu Thu Jan 27 20:35:50 2000 Return-Path: Received: from nic.funet.fi (nic.funet.fi [193.166.0.145]) by planet.fef.com (8.9.3/8.9.3) with ESMTP id UAA30547 for ; Thu, 27 Jan 2000 20:35:49 -0800 Received: from vger.rutgers.edu ([128.6.190.2]:27054 "EHLO vger.rutgers.edu" ident: "NO-IDENT-SERVICE[2]" smtp-auth: TLS-CIPHER: ) by nic.funet.fi with ESMTP id ; Fri, 28 Jan 2000 06:33:52 +0200 Received: by vger.rutgers.edu via listexpand id ; Thu, 27 Jan 2000 19:24:04 -0500 Received: by vger.rutgers.edu id ; Thu, 27 Jan 2000 19:23:31 -0500 Received: from zipcon.net ([209.221.136.5]:2794 "HELO zipcon.net") by vger.rutgers.edu with SMTP id ; Thu, 27 Jan 2000 19:20:42 -0500 Received: (qmail 1808 invoked from network); 28 Jan 2000 04:28:56 -0000 Received: from perseus.zipcon.net (HELO soggy.deldotd.com) (root@209.221.136.56) by mail.zipcon.net with SMTP; 28 Jan 2000 04:28:56 -0000 Received: by zipcon.net via sendmail from stdin id (Debian Smail3.2.0.102) for linux-raid@vger.rutgers.edu; Thu, 27 Jan 2000 20:18:40 -0800 (PST) Message-Id: Date: Thu, 27 Jan 2000 20:18:40 -0800 (PST) From: Brian Denheyer To: linux-list@ssc.com, linux-raid@vger.rutgers.edu Subject: how I set up raid-1 on an existing system (long) Mime-Version: 1.0 (generated by tm-edit 7.106) Content-Type: text/plain; charset=US-ASCII Sender: owner-linux-raid@vger.rutgers.edu Precedence: bulk X-Loop: majordomo@vger.rutgers.edu X-Orcpt: rfc822;linux-raid-outgoing Status: RO Hi here is my account of setting up raid-1 over an existing system. Setting up Raid-1 over an existing system Brian Denheyer briand@zipcon.net Much of this information can be found in both the "new" raid how-to http://ostenfeld.dk/~jakob/Software-RAID.HOWTO/ and the disk upgrade mini how-to. http://metalab.unc.edu/mdw/HOWTO/mini/Hard-Disk-Upgrade.html I personally find it very instructive to see real world examples with all of the trifling details, like where files are mounted, what commands are used to copy things, etc... I hope others who take the plunge will find this account of my experience helpful. I did make my mirror device the root partition, although I did not try to get things to boot from the mirror device. The how-to authors should feel free to take anything from here which might be useful or helpful and incorporate it into the how-to's. Caveats ------- If you do something wrong you could trash all the data on your harddisks. Make back-ups. Don't blame me if something goes wrong. Half the time I wasn't even sure it was going to work. But I have a 4GB DAT tape with all of my whole system on it, so I wasn't worried. Well, not too worried. Please be careful and think about what you're doing. I found the biggest problem to be file confusion. Getting hda/hdb/hdc mixed up. Modifying /etc/fstab when I meant to modify /new/etc/fstab and other such nonsense. So take it slow and double-check your work. Start ----- I started with 2 20.5GB IBM IDE disk drives. The BIOS was configured to autodetect the disk drives. This makes it easy to swap disks since you don't have to go into the BIOS set-up to redefine the disk type. Whenever you swap disks be sure and set those jumpers to master or slave as appropriate. It's also important to set the MODE of the disk drives to NORMAL and not to LARGE or LBA. I found that this made the geometry detection more uniform, i.e. the drives were detected with the same geometry. So now I have : /dev/hda : my original boot/root disk /dev/hdb : mirror drive 0 /dev/hdc : mirror drive 1 My first attempt ---------------- My first attempt resulted in a major problem. The boot process would always stop with errors on the mirror devices. Apparently the number of sectors in the superblock and physical number of sectors didn't agree. I believe the following may have caused this problem, but I don't really know. 1. The BIOS mode was set to something other than NORMAL which caused the disk geometries to be read differently even though the disks were the same. 2. My lack of understanding led me to create filesystems on the individual partitions (Don't do that !). So after much thought I decided to give it one more try and started from scratch. Essentially I followed the "new and improved" raid how-to. Patches ------- Since I didn't see any patches available for 2.2.14, I downgraded my kernel from 2.2.14 to 2.2.10. I then obtained the raid0145-19990724-2.2.10 patch file and applied the patch. I'm using a debian system and it currently provides the latest version of raidtools (0.90 version) as a package. You'll find raidtools2 in the unstable part of the distribution. apt-get install raidtools2 Sadly, many people do not use Debian and will need to fetch the source for the tools and build. It's very important that the boot scripts execute raidstart to set-up the mirror devices once they are up and running. After you have auto-detection working, you can remove those same scripts. Apply the patches and build raid support into the kernel. cd /usr/src/linux patch -p0 < raid0145-19990724-2.2.10 Rebuild the kernel being sure to add support for RAID-1 and raid auto-detection. Do the usual kernel build stuff and reboot so that you are running your shiny, new raid-aware kernel. I used cfdisk and partitioned the disks for use as : device mirror device mount-point ------ ------------- ----------- /dev/hd[bc]1 /dev/md0 / /dev/hd[bc]2 /dev/md1 /usr /dev/hd[bc]3 /dev/md2 /usr/local /dev/hd[bc]5 /dev/md3 /home Ans saved some space for BIG swap. /dev/hd[bc]6 swap Stupid mistake alert. Use "cfdisk /dev/hdb" NOT "cfdisk /dev/hdb1". I kept doing this by accident and couldn't figure out why cfdisk wouldn't start. Another Stupid mistake alert. Do NOT install a filesystem on the partitions you just created. Make Raid --------- I then set-up my /etc/raidtab file. Please see the end of this discussion for the file contents. IMPORTANT! Be sure and use a persistent-superblock setting of 1 so that the mirror will be useful the next time you boot. Make the partitions and then move on to the easy part. Make the raid devices by using mkraid. mkraid /dev/md0 mkraid /dev/md1 mkraid /dev/md2 mkraid /dev/md3 Here's a question : how do you speed the re-sync process ?? Re-nicing didn't seem to work and the big partitions take a LONG time. The amazing thing is that you can use the partitions while they are re-syncing. So I decided to install filesystems : mkfs.ext2 -b 4096 -R stride=4 /dev/md0 mkfs.ext2 -b 4096 -R stride=4 /dev/md1 mkfs.ext2 -b 4096 -R stride=4 /dev/md2 mkfs.ext2 -b 4096 -R stride=4 /dev/md3 I have found that the stride doesn't do didly for performance so you can omit the stride parameter when installing the filesystem on the mirror devices. To be safe I waited until the re-sync process finished. It takes forever... Just check /proc/mdstat every now and again to see how it's going. Once it was finished with the re-sync, do an fsck of the filesystems to make sure they're clean. Moving Files ------------ You then have to start moving files from your "old" system to the mirror devices. The "Hard Disk Upgrade Mini How-To" is VERY helpful in this regard. Read it carefully, especially the section about LILO. Here is an example. I want to transfer /usr/local to the mirror device which is destined for use as /usr/local. mkdir /newusrlocal mount /dev/md2 /newusrlocal cp -a /usr/local/* /newusrlocal umount /usr/local umount /newusrlocal mount /dev/md2 /usr/local /usr/local is now on the mirror device. Modify the /etc/fstab appropriately to get the filesystem mounted at boot-time. I then copied the rest of the files over, set-up my fstab appropriately so that the filesystems would be mounted on the appropriate mirror device and re-booted. PRESTO! Everything worked just fine. Hungry for challenge I turned my attention towards using the mirror device for my root filesystem. LILO : The Challenge -------------------- LILO scares me. I don't understand _exactly_ what it's doing. Whenever I change anything having to do with LILO I find myself holding my breath, hoping the system will boot the way I want. So it was no small leap of faith for me to attempt to use /dev/md0 as /. First things first. The partitions which are used for the mirror _must_ be marked as autodetected partitions (fd instead of 82) using cfdisk. Please be sure that the partitions are auto-detected properly _before_ you try to set-up the root partition on your mirrored device. Once the partitions are auto-deteced you will no longer need the raidstart stuff in the rc scripts to get raid started. WARNING: be sure and unmount filesystems and perform raidstop before you go and change the partition types ! You will be changing the partition types of the underlying ide partitions, i.e. hdb/c in my case. Be _sure_ and do a "umount /dev/md?" and "raidstop /dev/md?" _before_ you change the partition types. Change the types to fd on _all_ the partitions which are being used as mirror devices. Once I was convinced auto-detection was working I put a file system on /dev/md0, mounted it to /new, and then copied over the / files, being very careful not to copy /usr, /usr/local, /proc, etc... Once done, you have to be sure and create the directories you did not copy, like /proc for instance. WARNING: I use the hda drive to boot from, i.e. hdb/c are the mirrors. You are NOT going to boot from the mirror with this set-up. I'm too chicken, and I've been staying up too late as it is, to try patching LILO and booting from a mirrored device. Now the moment of truth. Here is the LILO file _exactly_ as I used it to create the new boot sequence. Please not that /boot is still on my original disk, i.e. /dev/hda : verbose=1 boot = /dev/hda # /sbin/lilo (the installer) copies the Lilo boot record # from the following file to the MBR location. install = /boot/boot.b # # I wrote a verbose boot menu. Lilo finds it here. message = /boot/message # The installer will build the following file. It tells # the boot-loader where the blocks of the kernels are. map = /boot/map compact prompt # Wait 5 seconds, then boot timeout = 50 image = /boot/vmlinuz-2.2.10 label = 2.2.10 root = /dev/hda1 read-only image = /boot/vmlinuz-2.2.10 label = m2.2.10 root = /dev/md0 read-only image = /boot/vmlinuz-2.2.14 label = 2.2.14 root = /dev/hda1 read-only Notice that the only change from what would be considered a "normal" config file, is the redirection of the root filesystem to /dev/md0. Please note that the _second_ image is the mirror root partition. So the default boot, and therefore the escape hatch, is to set-up /dev/hda1 as the / filesystem. If the mirrored filesystem doesn't work right, you can fall back to the original root filesystem to fix things. Modify the /new/etc/fstab file to reflect the fact that you will be putting / on /dev/md0. Don't forget to do this and don't modify /etc/fstab by accident !! Run lilo. Reboot. Wen the lilo prompt comes up, press tab and try using "m2.2.10". If you've done everything correctly, it will work. It did for me :-) If not, drop back to the original / location, on /dev/hda1, take a deep breath, and try to figure out what's screwed-up. Now the trickiy part. Once / is on /dev/md0, you need to modify lilo again to make /dev/md0 the root filesystem as the default case. Here's where my lack of LILO knowledge is really a problem. The above file will no longer work. LILO bails out with an error : "don't know how to handle device 0x0900" or some such nonsense. Here is what I did to get around this. It's probably not the correct way to do it, but it does work. Remember that /dev/hda1 still holds the original / filesystem. So create a home for it and mount it : mount /dev/hda1 /mnt Point all the files in the lilo.conf file to /mnt/boot, like so : verbose=1 boot = /dev/hda # /sbin/lilo (the installer) copies the Lilo boot record # from the following file to the MBR location. install = /mnt/boot/boot.b # # I wrote a verbose boot menu. Lilo finds it here. message = /mnt/boot/message # The installer will build the following file. It tells # the boot-loader where the blocks of the kernels are. map = /mnt/boot/map compact prompt # Wait 5 seconds, then boot timeout = 50 image = /mnt/boot/vmlinuz-2.2.10 label = m2.2.10 root = /dev/md0 read-only image = /mnt/boot/vmlinuz-2.2.10 label = 2.2.10 root = /dev/hda1 read-only image = /mnt/boot/vmlinuz-2.2.14 label = 2.2.14 root = /dev/hda1 read-only Notice that the order is switched so that /dev/md0 will be used for / by default. Now run lilo. It installs and the next time you boot, / will be on /dev/md0. Now go get some sleep. Here is a listing of my raidtab file I used to get things going. raiddev /dev/md0 raid-level 1 nr-raid-disks 2 nr-spare-disks 0 chunk-size 16 persistent-superblock 1 device /dev/hdb1 raid-disk 0 device /dev/hdc1 raid-disk 1 raiddev /dev/md1 raid-level 1 nr-raid-disks 2 nr-spare-disks 0 chunk-size 16 persistent-superblock 1 device /dev/hdb2 raid-disk 0 device /dev/hdc2 raid-disk 1 raiddev /dev/md2 raid-level 1 nr-raid-disks 2 nr-spare-disks 0 chunk-size 16 persistent-superblock 1 device /dev/hdb3 raid-disk 0 device /dev/hdc3 raid-disk 1 raiddev /dev/md3 raid-level 1 nr-raid-disks 2 nr-spare-disks 0 chunk-size 16 persistent-superblock 1 device /dev/hdb5 raid-disk 0 device /dev/hdc5 raid-disk 1