Repurposing Old Datto Server
I was able to aquire an old Siris 3 server from my old job. The motherboard it uses is an ASRock Rack D1541D4U-2T8R. Up until recently I've been using it for TrueNAS, but have been using another RAID card for this. Originally I was planning to just add a Mellanox CX311A card to have 10GB, but once I started looking up the specs of the server I realized a couple things. There an onboard RAID card that I had not bothered looking up before, since I assumed (stupid me) that I would not be able to flash it to IT mode. Well I was wrong; the RAID card it lists is LSI SAS 3008, which is apparently a rebranded LSI 9300-8i. This card will be 12Gb/s as opposed to my current 6Gb/s
- Setup TrueNAS on a temporary computer
- Reinstall TrueNAS on 2 NVMe drive mirror
- Downgrade RAM to 32GB
- Setup failover NIC
- Flash onboard 9300-8i to IT mode
- Install HP HSTNM-B017 SAS expander for external connectivity
- Verify "external" caddy drives work and are hot-swappable
- Future - upgrade SAS expander to 12Gb/s
- Future - Setup SSD Dataset
- Future - Setup
Setup TrueNAS on a temporary computer
I needed a temporary system to house my data while building out the new server. I just grabbed an old HP computer I had laying around and installed TrueNAS on it quick. For connecting the drives I had a spare H200 RAID controller from a Dell R710, already flashed to IT mode previously. I used my external drive caddy I had built previously to hold and power the drives; check farther down this article as I detail how I made that. I exported the settings from my current TrueNAS computer System Settings > General > Manage Configuration > Download File then powered down the system. I logged into the temporary TrueNAS and uploaded the settings in the same area, System Settings > General > Manage Configuration. After I rebooted, I needed to go to the console and change the IP to the new network interface.
- Select 1 Configure network interfaces
- Go into the old interface from the old server and remove the alias
- Go back and into the new interface (enp2s0 on mine) of the current HP computer and add the IP and CIDR to the aliasfield
- Save the new configuration, make sure it's working by going to the IP in your browser, and then make the changes permanent
- If desired go back into interfaces and remove the stale interface. I didn't bother since this is temporary.
I verified I could access the shares and needed to remount a few shares on my servers, but everything went smooth and I was back up and running in about 10 minutes. The next step I did a fresh install of TrueNAS.
Reinstall TrueNAS, downgrade RAM, Failover
In my previous configuration I had two SSD connected with SATA cables. In the future I may want to have a RAIDZ2 using six SSD so I decided to move the boot drive to two NVMe drives. I used two 256GB NVMe drives I had on hand. During setup just check both drives and it will set them up in a mirror. I also had decided to decrease the amount of RAM to repurpose the two 16GB sticks for my main Proxmox server. This brings it down to 32GB, which is still more than enough for my needs. When setting up IPMI I ran into some connectivity issues when trying to set a static IP, so I ended up just leaving it as DHCP and assigned a reservation for it.
I would like to have some kind of load balancing since there are two 10GB NICs, but right now I only have one ethernet SFP for my switch. I don't want it load balancing (if it would even work, doubt it) over a 10GB and 1GB link, so I'm going to set the 10GB NIC as primary and have it failover to the 1GB link.
3. Flash onboard 9300-8i to IT mode
With some googling, I found that the RAID card listed under this motherboard is actually the LSI 9300-8i. I started following the guide here but quickly found that I would need to take a few extra steps. I downloaded firmare version 16.00.12.00 instead, to avoid this issue. There also is no jumper to reset the onboard card, so when I tried writing the new firmware to the card, it failed. It would be less steps to just do everything in FreeDOS, but here is how I did it:
Flash firmware through EFI shell
I recommend ignoring anything about booting into the efi shell and just go to where I flashed via FreeDOS. I am detailing how I tried to do it via the uefi shell, failed that, and then successfully flashed through DOS.
First I searched here for the files needed. Search for "9300_8i P16" and I ended up downloading [zip]9300_8i_Package_P16_IR_IT_FW_BIOS_for_MSDOS_Windows. I had to search again for "sas3flash.efi" which led me to this; download the Installer_P14_for_UEFI.zip at the bottom. Again, this was kind of pointless since I ended up having to boot to FreeDOS to do most of it.
I formatted a thumb drive with NTFS and copied to the root of the drive SAS9300_8i_IT.bin, mptsas3.rom, and mpt3x64.rom from 9300_8i_Package_P16_IR_IT_FW_BIOS_for_MSDOS_Windows zip, and sas3flash.efi from Installer_P14_for_UEFI.zip. Plu in the thumb drive and boot the Siris 3. Press the F11 button at the prompt to bring up the boot menu, then boot to the uefi shell.
- Change to the thumb drive, mine was FS1
- "FS1:"
- Get SAS address, make sure to to a picture or write down
- "sas3flash.efi -list"
- It was at this point that I erased the flash on the card and tried writting the new firmware, which failed since I was not able to put the card in reset mode. After this happened I rebooted to make sure I didn't brick anything, and was able to use FreeDOS to finish things up
Flash firmware through FreeDOS
- Download FreeDOS and flash it to USB drive using Balena Etcher
- At the root of the drive, copy the same files as earlier: SAS9300_8i_IT.bin, mptsas3.rom, and mpt3x64.rom, and this time copy the DOS version of sasflash3, sas3flsh.exe. I renamed SAS9300_8i_IT.bin to 9300.bin, because FreeDOS can only have up to 8 characters in a filename.
- Boot up the Siris 3, press F11 to bring up the boot menu, and boot to FreeDOS.
- Use "DIR" to verify all files are there, then run:
- "sas3flsh.exe -list" write down/take a picture of the SAS address
- "sas3flsh.exe -o -e 7" to wipe the flash area
- "sas3flsh.exe -o -f 9300.bin -b mptsas3.rom" to flash the new firmware & bios
- "sas3flsh.exe -b mpt3x64.rom" to update uefi bsd version bios
- "sas3flsh.exe -o -sasadd XXXXXXXXXXXXXXX" change the X's to the SAS address from earlier
Install SAS expander, verify everything works
I have two HP HSTNM-B017 SAS expansion cards on hand. They are limited to 3Gb/s, so I may decide to upgrade in the future. For now, speed is not that important for what I use it for. The plan is to use the plan is to use one expansion card in the server to provide external connectivity through the SF-8088 port on the back to connect to the second expansion card, which I will be using with my external drive caddy. This will make it easy to add/remove drives for cold storage backups.
We use a SFF-8643 to SFF-8087 cable to plug into one of the ports on the mobo and plug into the Controller port of the expander.
