{"id":202,"date":"2022-07-03T18:50:56","date_gmt":"2022-07-03T15:50:56","guid":{"rendered":"http:\/\/vpv.kapsi.fi\/blog\/?p=202"},"modified":"2022-07-03T18:52:20","modified_gmt":"2022-07-03T15:52:20","slug":"sas2008-lba-seagate-ironwolfs-and-scary-log-messages","status":"publish","type":"post","link":"https:\/\/vpv.kapsi.fi\/blog\/2022\/07\/sas2008-lba-seagate-ironwolfs-and-scary-log-messages\/","title":{"rendered":"SAS2008 LBA, Seagate Ironwolfs and scary log messages"},"content":{"rendered":"\n<p>I built a home NAS two years ago, that was the first COVID summer and I finally had the time. It&#8217;s running Proxmox, which is running TrueNAS (then Core, now Scale) as a VM. An HBA card is passed directly to the TrueNAS VM. The HBA card is a Dell PERC H310, but I&#8217;ve crossflashed it so that now it shows up as an LSI SAS2008 PCI-Express Fusion-MPT SAS-2. The system originally had five ST4000VN008 disks (4 TB) in a RAIDZ2.<\/p>\n\n\n\n<p>Pretty much from the beginning I noticed the system was spewing out storage related error messages when booting up. ZFS also noticed, but after the TrueNAS VM was completely up, there were no more errors and I quite rarely rebooted or shut down the system, so I wasn&#8217;t too worried. The few read errors I got each boot I cleared with <code>zpool clear<\/code>, which probably was not the best idea.<\/p>\n\n\n\n<p>Last summer we had very cheap electricity here in Finland, something like 1-3 c\/kWh plus transfer and taxes. Well, this summer it can be even 60 c\/kWh during the worst times. I started shutting down my NAS when I knew we would not need it for a while. This made the disk issues worse.<\/p>\n\n\n\n<p><em>I know the high electricity prices are partly due to Russia&#8217;s attack in Ukraine and the sanctions against Russia. I completely support Ukraine, they are fighting for the freedom of all of the Eastern EU border states. <a href=\"https:\/\/www.defendukraine.org\/donate\">Please donate to support Ukraine.<\/a><\/em><\/p>\n\n\n\n<p>TrueNAS keeps only one day of systemd journal data (why?) so I&#8217;ve already lost the actual error messages. By going through my Google search history I was able to find some of the errors I got. They were like this:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">Unaligned partial completion ...\ntag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE ...\nprint_req_error: critical medium error ... <\/pre>\n\n\n\n<p>Because there&#8217;s quite a lot of discussion on the web about Ironwolf firmware issues, issues with NCQ etc. I hoped this was something that could be fixed with software. I tried passing many kernel options found by googling to the TrueNAS Scale kernel. I came up with <code>libata.force=noncq mpt3sas.msix_disable=1 mpt3sas.max_queue_depth=10000<\/code>. For more discussion on these issues, see <a href=\"https:\/\/www.reddit.com\/r\/synology\/comments\/eri5dg\/ds1019_ironwolf_8tb_softlocks_high_io_waits\/\">here<\/a>, <a href=\"https:\/\/www.truenas.com\/community\/threads\/synchronize-cache-command-timeout-error.55067\/\">here<\/a>, <a href=\"https:\/\/forums.servethehome.com\/index.php?threads\/lsi9211-8i-on-ubuntu-15-10-timeouts.8820\/page-2\">here<\/a>, <a href=\"https:\/\/forum.proxmox.com\/threads\/solved-proxmox-7-linux-5-11-22-issue-with-lsi-2008-controllers.93781\/page-2\">here<\/a>. <a href=\"https:\/\/blog.quindorian.org\/2019\/09\/ironwolf10tbfirmwarefix.html\/\">Seagate has actually released a firmware update from SC60 to SC61 for the larger Ironwolfs<\/a>, but I have the 4 TB ones without an update available.<\/p>\n\n\n\n<p>None of these options helped. Eventually the whole disk just disappeared. At this point it was clear to me that the issue was not a kernel bug, a disk firmware bug, an HBA firmware bug or anything like that. The disk had been faulty already on arrival.<\/p>\n\n\n\n<p>I noticed Seagate has come up with new versions of the Ironwolfs. The 4 TB version is now ST4000VN006 with 256 MB of cache instead of 64 MB. The new version is also physically thinner and might run cooler. I ordered one of those. Unfortunately the firmware version is still SC60.<\/p>\n\n\n\n<p>I replaced the faulty disk with the new one, ZFS resilvered the pool in about 8 hours and all is good again. I guess the moral of the story is that it seems like a disk could be defective, it probably is and you should start by replacing it.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I built a home NAS two years ago, that was the first COVID summer and I finally had the time. It&#8217;s running Proxmox, which is running TrueNAS (then Core, now Scale) as a VM. An HBA card is passed directly to the TrueNAS VM. The HBA card is a Dell PERC H310, but I&#8217;ve crossflashed [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[10,9],"tags":[],"class_list":["post-202","post","type-post","status-publish","format-standard","hentry","category-hardware","category-nas"],"_links":{"self":[{"href":"https:\/\/vpv.kapsi.fi\/blog\/wp-json\/wp\/v2\/posts\/202","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/vpv.kapsi.fi\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/vpv.kapsi.fi\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/vpv.kapsi.fi\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/vpv.kapsi.fi\/blog\/wp-json\/wp\/v2\/comments?post=202"}],"version-history":[{"count":1,"href":"https:\/\/vpv.kapsi.fi\/blog\/wp-json\/wp\/v2\/posts\/202\/revisions"}],"predecessor-version":[{"id":204,"href":"https:\/\/vpv.kapsi.fi\/blog\/wp-json\/wp\/v2\/posts\/202\/revisions\/204"}],"wp:attachment":[{"href":"https:\/\/vpv.kapsi.fi\/blog\/wp-json\/wp\/v2\/media?parent=202"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/vpv.kapsi.fi\/blog\/wp-json\/wp\/v2\/categories?post=202"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/vpv.kapsi.fi\/blog\/wp-json\/wp\/v2\/tags?post=202"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}