capablanca used to have two zpools: storage and archive. I recently combined both into one zpool with two mirror vdevs made out of 3 disks each (totalling 6 disks).
But its Supermicro mainboard doesn’t have enough onboard SATA ports to support 1 OS disk and 6 ZFS disks. I started looking for SATA PCIe expansion cards, and was disappointed that most were 1x - 4x. That doesn’t give full bandwidth to each disk. I bought the LSI 9211-8I because it uses 8x, and wasn’t too expensive.
Table of contents:
The LSI 9210-8I and 9211-8I default to IR (integrated RAID) mode, which means that the cards will implement hardware RAID. I wanted for my card to only provide more SATA ports so I needed to flash the firmware to IT (initiator target) mode.
Entering into the configuration menu during boot shows which firmware the card has:
In this case, 18.00.00.00-IR.
I first tried creating a FreeDOS boot usb so that I could follow the steps for flashing with DOS tools:
I tried running sas2flsh -listall, but saw:
ERROR: Failed to initialize PAL. Exiting Program.
Apparently this error means my BIOS is in EFI mode, doesn’t support something that the DOS tool needs, and that I need to flash with the EFI version of sas2flash.
I flashed the LSI card on another mainboard that didn’t have a built-in
EFI shell, so I had to prepare a USB with one. I chose to use
refind to create a USB stick that will EFI boot other things:
# apt install refind
(select no to avoid installing onto your computer :))
# refind-install --usedefault /dev/sdb1 --alldrivers
I copied the EFI version of sas2flash to the USB stick, and made a refind stanza to boot from it:
menuentry "sas2flash" {
loader /sas2flash.efi
}
Unfortunately this didn’t work.
What did work was to copy the tianocore repo’s Shell_Full.efi
into EFI/tools on the USB stick, and rename it to Shell.efi. It
showed up in the tools section of refind, and booted without incident.
From there, switch to the USB stick’s filesystem by enumerating fsN:,
and doing a dir to see which files are present:
From here, sas2flash.efi -listall should work:
And flashing worked no problem. The commands to use are:
sas2flash.efi -o -e 6
sas2flash.efi -o -f 2108it.bin -b x64sas2.rom
Note that x64sas2.rom is shown in these screenshots.
This didn’t actually work for me, I had to flash mptsas2.rom. Here are
the files I used (downloaded from Broadcom’s sas-9210-8i page):
Installer_P20_for_UEFI gives /sas2flash_efi_ebc_rel/sas2flash.efi9210-8i_Package_P20_IR_IT_FW_BIOS_for_MSDOS_Windows gives sasbios_rel/mptsas2.rom9210-8i_Package_P20_IR_IT_FW_BIOS_for_MSDOS_Windows gives Firmware/HBA_9210_8i_IT/2108it.binWhen I flashed x64sas2.rom, it said “No image” under x86 BIOS in the
-listall output:
And the LSI boot sequence didn’t start when the machine booted.
After flashing mptsas2.rom, there was an entry for x86 BIOS:
And the LSI boot sequence started again. The configuration menu now showed the updated firmware, and that it was in IT mode:
First, add a GPT and a single partition aligned to a 4k boundary:
# gpart create -s gpt /dev/diskid/DISK-WD-WCC4M2UHLZ54
diskid/DISK-WD-WCC4M2UHLZ54 created
# gpart add -a 4k -t freebsd-zfs -l WCC4M2UHLZ54p1 /dev/diskid/DISK-WD-WCC4M2UHLZ54
diskid/DISK-WD-WCC4M2UHLZ54p1 added
# geli init -s 4096 -l 256 /dev/gpt/WCC4M2UHLZ54p1
Enter new passphrase:
Reenter new passphrase:
Metadata backup can be found in /var/backups/gpt_WCC4M2UHLZ54p1.eli and
can be restored with the following command:
# geli restore /var/backups/gpt_WCC4M2UHLZ54p1.eli /dev/gpt/WCC4M2UHLZ54p1
Before, my archive zpool looked like this:
pool: archive
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
archive ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
gpt/WMC300563174p1.eli ONLINE 0 0 0
gpt/WCC4M2773993p1.eli ONLINE 0 0 0
gpt/WMAZA0686209p1.eli ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
gpt/WMAZA9460416p1.eli ONLINE 0 0 0
gpt/WMC300578369p1.eli ONLINE 0 0 0
Add to the unbalanced mirror-1 with:
# geli attach gpt/WCC4M2UHLZ54p1
# zpool attach archive gpt/WMC300578369p1.eli gpt/WCC4M2UHLZ54p1.eli
# zpool status archive
pool: archive
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Thu Nov 9 21:29:00 2017
146G scanned out of 2.96T at 1.82G/s, 0h26m to go
113M resilvered, 4.80% done
config:
NAME STATE READ WRITE CKSUM
archive ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
gpt/WMC300563174p1.eli ONLINE 0 0 0
gpt/WCC4M2773993p1.eli ONLINE 0 0 0
gpt/WMAZA0686209p1.eli ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
gpt/WMAZA9460416p1.eli ONLINE 0 0 0
gpt/WMC300578369p1.eli ONLINE 0 0 0
gpt/WCC4M2UHLZ54p1.eli ONLINE 0 0 0 (resilvering)
The resilver completed after 4h42m. /archive/ uses 2.96T according to
the zpool status above. If only the data is resilvered, the max write
bandwidth to the new disk is:
>>> mb = 2.96*(1024**2)
>>> seconds = (4*60+24)*60
>>> mb/seconds
195.9460202020202
only roughly 196 MB/s. According to zorinaq.com’s blog post, the 9211-8i should get “300-350MB/s/port”. If 196 MB/s is the correct measure, it means that the GELI and ZFS overhead is potentially cutting the bandwidth to the disk in half (!), assuming that the read bandwidth from the other two disks in mirror-1 is not the limiting factor.
Unfortunately I didn’t measure the raw bandwidth to the disk before attaching it to mirror-1.
However, the read bandwidth of 1.82G/s means that there are parallel reads off each disk in mirror-1 (assuming only mirror-1 is participating in the resilver). Splitting the read bandwidth by 3:
>>> 1.82*1000/3
606.6666666666666
This is very close to the theoretical max of 6Gbps SATA at 600 MB/s.