How to install FreeBSD 8.0 using only ZFS partitions? (Newbie proof, I hope)

Ok, so I want to install the newly released FreeBSD 8.0 and only have ZFS partitions.

Questions I had that I am answering here for you:

  1. Does Sysinstall support ZFS?

    No. You have to install manually using Fixit.

  2. Can I boot off a ZFS partition?

    Yes, you can. Takes some effort to set it up though.

  3. Is it easy?

    No, it isn’t easy it is hard. Unless you have used FreeBSD for some time, then it is just tedious and not really easy or hard.

  4. Is there a guide?

    Yes, it is here:
    http://wiki.freebsd.org/RootOnZFS/ZFSBootPartition
    But hopefully by the time I am done, the best guide will be this post.

Ok, so here it goes, I am following this guide: http://wiki.freebsd.org/RootOnZFS/ZFSBootPartition
I am going to improve upon it and try to make it newbie proof and not skip steps in my guide when this guide skips steps. Why am I making it newbie proof? I don’t know, you would think if you are doing ZFS you aren’t a newbie, but who knows. Better safe than sorry.

Steps for How to install FreeBSD 8.0 using only ZFS partitions?
Step 1. Creating a bootable ZFS Filesystem

  1. Download the FreeBSD 8 DVD ISO from here: http://www.freebsd.org/where.html
    Ok, so this isn’t exactly a download link, but it takes you to where you choose your architecture. Click on the [ISO] link next to your Architecture type.
    If you are a newbie and don’t know your architecture, you probably want i386. If you just bought a new 64 bit machine then you want amd64.
    Ok, so the actually link you want is the one that looks as follows:
    For amd64 – 8.0-RELEASE-amd64-dvd1.iso.gz
    For i386 – 8.0-RELEASE-i386-dvd1.iso.gz

  2. Extract it as it is zipped.
  3. Burn the ISO to DVD disk (or if you are using VMWare just point your VM’s CD-ROM at the ISO).
  4. Boot off the FreeBSD 8 DVD and let it boot all the way up.
  5. Choose your country/region.
    You should now be at the Sysinstall Main Menu. Remember, we cannot use Sysinstall because it doesn’t yet support ZFS. (I am hoping for a new installer over fixing this old one.)

  6. Choose Fixit. You will be promted to Please choose a fixit option.
  7. Choose CDROM/DVD.
    You are now at a Fixit command prompt. And if you are following the wiki guide, you are only at that guide’s step 3. Create MBR disk

  8. Create an MBR disk by running the following command:
    gpart create -s mbr ad0

    However, what the guide assumes you already know is that ad0 is the name of your hard drive and can be different for each installation. I am installing on VMWare using SCSI and the name of my hard drive is da0. So I would run:

    gpart create -s mbr da0

    You can find out your by looking at a directory listing of /dev if you are familiar with common names, otherwise, you can start a Standard install and see what name is used when you get to the Disk label editor section of the install.

    I am going to use da0 or ad0 in the rest of my document interchangeably, so you need to remember to always use the correct disk name for your drive.

  9. Run the following command to show the information for your disk
    gpart show ad0

    There are two columns of numbers. The first column number is a start block for the slice and the second is the size in blocks for the slice. 63 is the start of every first slice. However, the end of a slice is variable depending on the size of your hard drive.

    A slice is not exactly a partition. On FreeBSD you first create a Slice and then you create your partitions on your slice. The next steps will show you have to do both.

  10. Create a new Slice using the entire disk.

    Obviously the guy who wrote the wiki I am following already had two slices for windows, so he must have been dual booting. I am assuming that you are not dual booting and that you are installing a new server and you plan to use the entire disk.

    To create a slice using the entire disk, run this command replacing the value after the -s with the value you saw when you ran the previous command.

    gpart add -b 63 -s 104857515 -t freebsd da0

    It will output something like “da0s1 added”. If you look at the da0s1 string, it makes sense. da0 (your disk name) s1 (slice 1 – the slice you just created).

  11. Now lets create our slice. No, I am not sure why you have to both add the slice and create the slice, but I am sure there is a good reason.
    gpart create -s BSD da0s1
  12. Lets make our slice the active slice. Sounds like this is only sometimes needed. Better to do it and not need it than to not do it an find out you need it.
    gpart set -a active -i 1 da0

    You can run the gpart show da0 command again to make sure it is set as active.

  13. Look at the slice.
    gpart show da0s1

    Again, you will have two rows of numbers. This time the first number is 0 and the second number is the size of the slice.

    We want at least two partitions, one for / and one as a swap partition. So we need to determine how much space we want for the swap partition. I want 3 GB.

    Now we have to convert the desired size from GB to sectors.

    1 kilobyte = 1024 bytes or 2 sectors (Sectors are normally 512 Bytes)
    1 megabyte = 1024 kilobytes
    1 gigabyte = 1024 megabytes

    So to get the number of sectors in 1 GB, we need to use the following equation:

    Gigabytes in Sectors = NumberOfGB * numberOfMBInAGB * NumberOfKBInAMB * NumberOfSectorsInAKB

    1 GB in sectors = 1 * 1024 * 1024 * 2 = 2097152
    3 GB in sectors = 3 * 1024 * 1024 * 2 = 6291456

    So take the total size of your slice in sectors and subtract 6291456 and you will have the size of your / partition. And our swap partition will be 6291456.

  14. Create your / partition.
    gpart add -i 1 -b 0 -s 98566059 -t freebsd-zfs da0s1
  15. Create the swap partition.
    gpart add -i 2 -b 98566059 -s 6291456 -t freebsd-swap da0s1
  16. Load the ZFS kernel module.
    kldload /mnt2/boot/kernel/opensolaris.ko
    kldload /mnt2/boot/kernel/zfs.ko
  17. Create your zpools.
    mkdir /boot/zfs
    zpool create zroot /dev/da0s1a
    zpool set bootfs=zroot zroot
  18. Install the boot manager.
    gpart bootcode -b /mnt2/boot/boot0 da0
  19. Install ZFS boot.
    zpool export zroot
    dd if=/mnt2/boot/zfsboot of=/dev/da0s1 count=1
    dd if=/mnt2/boot/zfsboot of=/dev/da0s1a skip=1 seek=1024
    zpool import zroot

Yeah we are done with step 1. Stay tuned for step 2 and step 3 coming.

… I am back for round 2…er uh…step 2 that is.

Step 2. Installing FreeBSD to the ZFS filesystem

Ok for those of you who skipped the details above, I am reading this wiki:
http://wiki.freebsd.org/RootOnZFS/ZFSBootPartition

My intention is to make a more thorough and newbie proof version of this wiki. So here we go, diving into step 2.

  1. Create the ZFS hierarchy.

    Wow, this is going to be a lot of tedious typing. You know, while FreeBSD didn’t make an installer for all this, how hard would it have been to create a couple of scripts and include them on the CD so this would be easier.

    zfs set checksum=fletcher4 zroot

    zfs create -o compression=on -o exec=on -o setuid=off zroot/tmp
    chmod 1777 /zroot/tmp

    zfs create zroot/usr
    zfs create zroot/usr/home
    cd zroot ; ln -s /usr/home home

    zfs create -o compression=lzjb -o setuid=off zroot/usr/ports
    zfs create -o compression=off -o exec=off -o setuid=off zroot/usr/ports/distfiles
    zfs create -o compression=off -o exec=off -o setuid=off zroot/usr/ports/packages

    zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/usr/src

    zfs create zroot/var
    zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/crash
    zfs create -o exec=off -o setuid=off zroot/var/db
    zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/db/pkg
    zfs create -o exec=off -o setuid=off zroot/var/empty
    zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/log
    zfs create -o compression=gzip -o exec=off -o setuid=off zroot/var/mail
    zfs create -o exec=off -o setuid=off zroot/var/run
    zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/tmp
    chmod 1777 /zroot/var/tmp

    cd /dist/8.0-RELEASE
    export DESTDIR=/zroot
    for dir in base catpages dict doc games info lib32 manpages ports; \
    do (cd $dir ; ./install.sh) ; done
    cd src ; ./install.sh all
    cd ../kernels ; ./install.sh generic
    cd /zroot/boot ; cp -Rlp GENERIC/* /zroot/boot/kernel/
    zfs set readonly=on zroot/var/empty

    chroot /zroot

    echo ‘zfs_enable=”YES”‘ > /etc/rc.conf
    echo ‘hostname=”zfs.mydomain.local”‘ >> /etc/rc.conf
    echo ‘ifconfig_em0=”DHCP”‘ >> /etc/rc.conf

    echo ‘zfs_load=”YES”‘ > /boot/loader.conf
    echo ‘vfs.root.mountfrom=”zfs:zroot”‘ >> /boot/loader.conf

    echo ‘LOADER_ZFS_SUPPORT=YES’ > /etc/src.conf

    mount -t devfs devfs /dev
    export DESTDIR=””

    cd /usr/src/sys/boot/
    make obj
    make depend
    make
    cd i386/loader
    make install

    passwd

    tzsetup

    cd /etc/mail
    make aliases
    umount /dev
    exit
    cp /boot/zfs/zpool.cache /zroot/boot/zfs/zpool.cache

Warning! There is only one line that might catch a newbie off-guard. Every other line you can type in as is but this one.

echo ‘ifconfig_em0=”DHCP”‘ >> /etc/rc.conf

On FreeBSD this is how you setup your network card to use FreeBSD. However, while my card is em0, not all cards are em0. Run the ifconfig command on FreeBSD to see your card type and replace em0 with the type for you card.

Step 3 – Finish

I followed the guide almost exactly except I had to do a cd / before unmounting. So I added that command where it needs to be, so this should be very newbie proof.

  1. Run these commands.
    cat < /zroot/etc/fstab # Device Mountpoint FStype Options Dump Pass# /dev/ad0s3b none swap sw 0 0 EOF export LD_LIBRARY_PATH=/mnt2/lib cd / zfs unmount -a zfs set mountpoint=legacy zroot zfs set mountpoint=/tmp zroot/tmp zfs set mountpoint=/usr zroot/usr zfs set mountpoint=/var zroot/var

I made some mistakes but finally got it to work.

38 Comments

  1. Some really select content on this website, saved to fav.

    my blog post :: sell music online (liverpoolsu.com)

  2. autozone auto parts online store truck parts

    How to install FreeBSD 8.0 using only ZFS partitions? (Newbie proof, I hope) | Rhyous

  3. john lewis advert girl growing up

    How to install FreeBSD 8.0 using only ZFS partitions? (Newbie proof, I hope) | Rhyous

  4. Anderson says:

    So to conclude about the Casio Gshock Mudman Solar Gw9000A-1.
    In simple terms, you might be not likely to find another
    line of watches that keeps to the same standard.

    Casio has always been into high quality electronic goods.

  5. fbenter.net says:

    You need to combine practicality and personal touch in the gifts.

    People who tend to overpack their personal
    effects are always caught up in this dilemma. Fortunately, these days
    baby bags come in manly colors and designs.

  6. jh says:

    This didn't work for me. I didn't understand the usage of "cat" in the script (nor the cryptic comments by users; my experience and experiments with the cat command tell me it can't work) and replaced that part by creating the (nonexistent) fstab file with two echo... lines.

    Booting resulted in:
    F1 FreeBSD

    F6 PXE
    Boot: F1

    but no further progress.
    Thanks anyway, jh

    • varj says:

      i get this far too, it seems somehow the bootloader isn't correctly installed. (after "F1 FREEBSD F6 PXE", you're supposed to get the list with boot normal, single user mode, etc to the left and the FreeBSD logo on the right, right?)

  7. SIFE says:

    i followed all steps with no error, but in end I can't boot the system with no error showing, I test this in VirtualBox 3.2.12 with 8.2-RELEASE-AMD64.
    To those didn't find lib32 directory, because this directory available only in AMD64 and IA64 I guess.

  8. Kengaru says:

    Great howto!

    May could help someone, or improve this post:

    Unfortunately, trying to install the 8.1-RELEASE got no ATA DVD drive in old PC whose have to be the home router and NAS. Also, no USB boot at BIOS. So, it's impossible to boot from DVD or memstick image.

    Booted from Live FS CDROM, then, after creating pool (RAIDZ1) and ZFS hierarchy, tried to mount the memstick image on my 1Gb USB drive (/dev/da0a) to any dir under /mnt, then started extracting distributions. Kernel panic - low kmem, I have only 512Mb RAM on the box.

    We brake for nobody!

    So, next boot we set vm.kmem_size="330M"
    vm.kmem_size_max="330M"
    vfs.zfs.arc_max="40M"
    vfs.zfs.vdev.cache.size="5M" at the loader prompt and then boot. (Don't forget to place that to the /boot/loader.conf).

    And, after boot and diving to fixit shell, zpool import -f zroot to get where we were when crashed. Then, followed your manual, all goes like a charm.

    Thanks a lot.

  9. mike says:

    Ok, there is now an actually easy way of installing FreeBSD root on ZFS using mfsbsd livecd. I did a complete write up on my blog: http://mpietruszka.com/2010/08/14/freebsd-on-zfs-update-1/, but all you really need to get it going is just the mfsbsd livecd and two commands:

    Mount the cd:
    mount_cd9660 /dev/acd0 /cdrom
    Install ZFS version 15/FreebSD:
    zfsinstall -d /dev/ad0 -t /cdrom/8.1-RELEASE-amd64.tar.xz -s 4G -V 15

    Just to let you know, pjd has started testing ZFS version 28 so we might see that trickle down real soon as well.

  10. Tuco says:

    Not exactly the same, but Debian GNU/kFreeBSD has an installer with ZFS support: http://robertmh.wordpress.com/2010/09/06/debian-installer-with-zfs/

  11. dmitri says:

    That should be: cat < /zroot/etc/fstab as in the docs. Sorry.

  12. dmitri says:

    Thanks! Worked like a charm with 8.1
    According to the docs, installing the bootloader (the steps "cd /usr/src/sys/boot/" ... make, et c.,)
    are not required because 8.1 ships with a zfs aware bootloader
    However, it tried it just to see, and it does no harm.

    To n00b-proof your script, you might want to change:
    "cat < /zroot/etc/fstab" to
    "cat < /zroot/etc/fstab" _I_ know that ^D will get me an EOF, but someone just typing the script might get 'EOF' as the last line of /etc/fstab which might result in some interesting, though harmless, errors.

    Once again, thanks for a really great post. You rock, and now so does my Beastie.

  13. vincent says:

    nice job, many thks ... just one question, I tried with success a slight updated version of this how-to with a mirrored zpool but I try now to add a second (non mirrored) zpool on the same disk, ... is there a way to do this ?

  14. mike says:

    I would like to second what the poster above me (ben) has said. I do receive the same error messages upon boot.

  15. ben says:

    ...I should also add, one odd thing: my disk device was called ad3 (not da0, or ad0, or whatever). I partitioned it correctly (after some messing around), and the above commands all worked perfectly and gave the expected output. I had a slice for the full disk, partition one was 180-something GB (200GB disk) and then 4GB of swap; all the zfs creation commands worked; all fine.
    I have only one HDD, a DVD burner (internal), and that's it.

  16. ben says:

    Full disclosure: I have used linux for some time, but I am completely new to FreeBSD. Wanted to try ZFS natively for a project I'm working on.

    I followed your guide and everything worked smoothly - however when I reboot, I get the following after the kernel loads/recognises devices:

    ...
    uhub4: 8 ports with 8 removable, self powered
    Root mount waiting for: usbus4
    Trying to mount root from zfs:zroot
    ROOT MOUNT ERROR:
    If you have invalid mount options, reboot, and first try the following from the loader prompt:

    set vfs.root.mountfrom.options=rw

    and then remove invalid mount options from /etc/fstab.

    Loader variables:
    vfs.root.mountfrom=zfs:zroot
    vfs.root.mountfrom.options=

    Manual root filesystem specification:
    : Mount using filesystem
    eg. ufs:/dev/da0s1a
    eg. cd9660:/dev/acd0
    This is equivalent to: mount -t cd9660 /dev/acd0 /

    ? List valid disk boot devices
    Abort manual input

    My keyboard (a logitech usb one with backlit keys) doesn't light up and is inoperable at this point. I thought the 'uhub' stuff might be referring to the kb, and it was ruining the boot process, so I disconnected it (and the usb mouse) - no effect, same error. uhub/usbus stuff still comes up at the end of the output preceding the boot failure.

    I also tried booting up, entering option 6 to go into bootloader prompt, followed the system's advice and set vfs.root.mountfrom.options=rw - this does nothing, it just shows that option as having the value 'rw' when you boot up and the process fails again in exactly the same way.

    I have no idea what's going on, can anyone help me?
    I was thinking perhaps I wrote information to fstab incorrectly, (it was the only file I didn't check with a 'cat /etc/fstab' afterr creating it, while writing out the commands in the tutorial), but that only has an entry for 'swap', so I don't know how this could affect things.
    I thoroughly checked and I believe I wrote out every other command in the tutorial correctly.

    Really disappointed this didn't work for me, and appears this hasn't happened to anyone else - can someone tell me what might be wrong?

  17. David says:

    Possibly a faulty DVD? I have lib32 among others under /8.0-RELEASE on my amd64 DVD.

  18. Frank says:

    Thanks for the info david, but when I cd to the /dist/8.0 on the CD I did a ls command I found all the directories base, catpages, dict, doc, games info, manpages ports except for lib32. and I'm using 8.0-RELEASE-i386-dvd1.iso downloaded from freebsd

  19. David says:

    Frank, the lib32 error could occur if you made a typo or omitted lib32 when typing this line:

    for dir in base catpages dict doc games info lib32 manpages ports; \

    Whatever the reason, the installer is complaining that there's no lib32 directory.

    BTW I've also found this other howto very helpful for ZFS root setups:

    http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/RAIDZ1

  20. Frank says:

    Hi,
    Awesome post, I have some questions, after cd it to //8.0-release and did the
    do (cd $dir; ./install.sh) ; done
    I got some error messages pop up saying
    can't cd to lib32
    ./install.sh: not found
    anyone encounter this?

  21. [...] following  How to install FreeBSD 8.0 using only ZFS partitions numerous times to install FreeBSD on ZFS I decided automate things.  So I created a pair of shell [...]

  22. [...] How to install FreeBSD 8.0 using only ZFS partitions is a step-by-step guide for install FreeBSD 8.0-Release on to ZFS at rhyous.com that works. [...]

  23. Slava says:

    bah...looks like you did have the right command but wordpress filters it and messes it up...some of it got cut off. It just did the same thing in my previous commend. The command that's pasted here currently doesn't let the user input the required lines and create the file...for people running into the same issue with fstab, the full command is on step 3 in http://wiki.freebsd.org/RootOnZFS/ZFSBootPartition

  24. Slava says:

    Hi, great post, thanks for the detailed instruction! I've run into a problem though; on Step 3 when I run 'cat < /zroot/etc/fstab', it says the file does not exist.

    I unmounted everything and reboot; I get the boot loader but when I try to start FreeBSD, it fails to mount zfs:zroot and leaves me on the manual root mount prompt.

    I can see all the files on the file system through the boot loader command line, so everything seems to be intact, I'm guessing it's just the missing /etc/fstab that's causing the problem. Can you please tell me how I can fix this? I've looked over all the steps multiple times and can't find where I made my mistake.

    Thank you

    Slava

    • Slava says:

      OK, I got it...the command you have posted up there didn't work for me, I needed to use the command from the other tutorial 'cat < /zroot/etc/fstab'

      Also, after 'cd zroot ; ln -s /usr/home home', it keeps you in the zroot directory. So you should add a cd.. or cd / to go back to /. Not sure if that's required, but I wanted to make sure I ran everything from / unless otherwise specified.

      My FreeBSD 8 system boots in ZFS now, thank you!!

      Slava

  25. David says:

    OK, I think I got this working. Mirroring two disks in the zroot partition involves the following:

    1. Follow all steps above.

    2. On the second disk (da1 in my case) follow steps 9-14 above. Allocate the entire disk to the root partition. (Also, don't follow step 8; you do NOT install an MBR on the second disk.)

    3. Run this command to add the second disk to the zroot pool:

    zpool attach zroot da0s1a da1s1a

    4. Verify that you now have two disks in zroot:

    zpool status zroot

    That's it.

  26. David says:

    1. Yup, that appears to be the case. The machine is clearly running an amd64 kernel. Thanks for the sanity check.

    2. Yes, I have multiple disks in this machine. I would like to mirror the first two of them for the zroot pool.

    Is this doable using the method described here, and if so how?

    thanks again

  27. David says:

    Worked for me on 8.0-RELEASE amd64, thanks.

    Two questions:

    1. In step 2.1, there is no 'amd64' directory under /usr/src/sys/boot. Is i386 OK?

    2. ZFS newbie question: How to add disks to this zroot pool?

    The command 'zpool add zroot da1' returns this error:

    cannot add to 'zroot': root pool can not have multiple vdevs or separate logs

    Thanks again.

    • rhyous says:

      1. Thanks for pointing that out. I believe I did using i386 and it was fine. Maybe they haven't made amd64 boot environment because i386 boot environment can boot and am64 machine.

      2. Are you trying to add multiple physical disks and you are getting this error? I am not sure, I only had one disk when I did this.

  28. Brandon says:

    Thank you! I've been using FreeBSD since 3.1 but still wouldn't have ever gotten it to work without your instructions

  29. Greg says:

    Great instructions...worked well for me. The line at the end for fstab reads...

    /dev/ad0s3b none swap sw 0 0

    It should be...

    /dev/ad0s1b none swap sw 0 0

    This isn't instantly obviously since it still boots ok...but it won't use a swap partition unless you give it the correct device name.

  30. Igor Galić says:

    You can use qualifiers such as M or G in gpart's size parameter. That should be easier to read for you, and it's up to gpart to do the messy calculations.
    (I would only for the last partition specify the size in sectors, so you fill it all up)

  31. Russ says:

    Wow, perfect! Thank you. I've "messed" with FreeBSD a few times over the years, but I am really needing something with ZFS support, and OpenSolaris is, uh, well, not ok for my needs, so FreeBSD it is. So, newb to FreeBSD, but been using Unix (and then Linux) for more than 20 years, so this post is EXACTLY what I needed. Again, thank you very much!

Leave a Reply

How to post code in comments?