The more I listen to the guys at BSDNow the more I start thinking about using BSD/JAILS/ZFS to solve the IT related problems that I face.
The current problem surfaced in our Advanced Optical Imaging Research Group: there was a need for a storage with fault tolerance on disk fails, and it had to be accessible around the department with Windows/Linux, and also from home.
+ it had to be cheap/free
+ low maintenance cost (time, $)
+ Gigabit fast on reads
+ TB largeThere were several options from which we could choose:
- Cloud based solutions: Google Drive, Dropbox – we already use these for personal storages, but for raw measurement and simulation data it would cost us too much to share with each other on the long run
- Western Digital NAS/Apple Time Machine – expensive or no redundancy, no access to the software
- Use a USB disk storage – no redundancy, not fast enough, not networked
- Shared folder on the LAB PC – no redundancy, high CPU usage
- Custom hw/sw: dedicate an old desktop PC for the job with a linux server distro – a good choice
- Use server hardware with Linux/BSD – an even better choice
Luckily for us, my friend from kindergarten had access to obsolete servers so I asked if their department (in Saarbrücken) was considering selling some old stuff. The stuff I got:
As I wanted to try out something new so I have chosen FreeNAS 9.2, which is based on FreeBSD. The fun part is, that it is fully configurable via a web page. The OS itself sits on a 8 GB pendrive, uses the Ram as a cache and the disks as rw storage. This is the recommended setup. Don’t choose a bigger stick or an HDD, because the image will use up the full drive.
The setup was as easy as writing a disk image to a pendrive. After plugging in the memory stick it will boot up with a web server and after getting the ip the rest can be configured from a browser. IT IS EASY, and they have a good web documentation. The boot process is slow, but you don’t want to restart it every day. Here is good youtube video from iXSystems. (Ad: If you want a custom built NAS specifically for your needs, just order from them, they will ask every question, and you will get a TrueNAS.)
Some tips when planning the HW requirements:
- ZFS needs Ram for good speed: 8 GB + 1 GB for every TB of storage
- A two core cpu can be enough if you use a hardware ethernet card. If you are planning to use other features, chose a 4 core setup.
- Use 4 or more disks.
- Don’t use hardware raid, it will defeat ZFS disk management.
- A 4 Gb pendrive for the os. 8 GB for 9.3 where you can switch boot environments after upgrades.
- Enable L4Z compressing on the whole volume, which can lower disk IO and speed up access.
- Raid 10 is a good option, but if you have more than 6 disks and you don’t need the extra write speeds, choose a RaidZ2 setup.
- You must build your ZFS storage first, then set up a home directory, and after that you can set up the users with their home directory pointing to the new ZFS folder
I won’t go into more details, just try it. Here is the list of what functions we use:
- Virtualbox Jail – we had a small Windows desktop that had to be accessed globally. The jail comes with the phpvirtualbox web page, and it is very useful. The only thing is missing is USB management.
- ZFS snapshots – every shared folder is snapshotted hourly and they are kept for two weeks. So if a file is deleted accidentally, it can be restored for two weeks. The space usage is minimum, and it is only a problem when something is deleted.
- ZFS compression – 2.5 TB is the total disk space, with around 1TB data written already to the storage with around 1,20-1,35 compression rate.
With one NAS we solved the ageing disk problem also, so the money to upgrade our desktops can be spent on research equipment. As the local disks are backed up, we can still use the 5+ year hardware.
After the first year:
- With FreeNAS 9.3 the USB uses zfs also, which means snapshots, and selectable boot environments.
- Our 6 disk RaidZ2 setup can saturate both 1G Ethernet networks, and all the clients are happy.
- I started using ZFS on Linux with ubuntu on a 1 CPU/3 GB RAM PC. The memory usage is low enough that I can run two virtual machines, and no freezing.