TrueNAS - slow sequential reads on Windows, fine on Mac OS

Jump to latest Follow Reply
H

hestermofet

Ars Legatus Legionis
24,887
  • #1
    I have a TrueNAS box that worked fine for years but now it's acting weird. 128 GB of RAM so that's likely not the issue. Specs:
    2x 6 core Xeon Westmere or something. Quite old Dell R710.
    128 GB ECC DDR3
    5x 14TB Seagate Exos X16 in RAIDZ1
    OS drive is a Kingston SATA SSD
    Mellanox Connect-X3 10GbE
    Startech 8x SATA HBA connected to the Dell backplane
    TP Link 10GbE switches (multiple models) with 10GbE uplink, but clients are connected with Gigabit

    I get 110 MB/s RW in Mac Catalina on a 2011 Mac Mini. I've tried a few Windows boxes, and they all max out at 40 MB/s reads, but writes seem fine. Around 110 MB/s. Random performance seems great, it's just sequential which is slow. I'm using CrystalDiskMark, AmorphousDiskMark, and Windows Explorer file copy to test.

    Everything I've tried is Windows 11. I haven't had a chance to test performance in Linux yet as nothing it's installed on has Ethernet, just WiFi. Don't have Windows 10 installed anywhere to test. I've tried speed tests in multiple parts of the house connected by different Ethernet cabling/backhaul and switches, and same/similar results so it's likely not a networking issue, probably a Windows software configuration issue.

    I've tried turning off atime but it only improved things from 30 MB/s to 40 MB/s. Since there WAS an improvement there, there might be multiple things going on. Any ideas on where to start to troubleshoot?
     
    B

    brendan_kearney

    Ars Tribunus Militum
    2,076
  • #2
    if you turned off atime (with noatime in /etc/fstab), then also add nodiratime and noexec (this more for security), as well. beware journaling (and some recoverability) will be diminished with all of this, but backups, right...

    can you capture and analyze the traffic? are the windows boxes using older versions of SMB? how well tuned is the SMB server? Mac has be persnickety in my experience and tuning helps with performance.

    some general tweaks i found for performance:
    Code: 
            socket options = IPTOS_LOWDELAY TCP_NODELAY
        min receivefile size = 2048
        use sendfile = true
        aio read size = 2048
        aio write size = 2048
        #write cache size = 2048000
        read raw = yes
        write raw = yes
        getwd cache = yes
        oplocks = yes
        max xmit = 32768
        dead time = 15
        large readwrite = yes
    some mac specific tweaks, too:
    Code: 
            #suggested settings for iThings...
        #min protocol = SMB2_02
        ea support = yes
        #vfs objects = catia fruit recycle streams_xattr
        fruit:appl = yes
        fruit:copyfile = yes
        fruit:metadata = stream
        fruit:model = MacSamba
        fruit:posix_rename = yes
        fruit:veto_appledouble = no
        fruit:wipe_intentionally_left_blank_rfork = yes
        fruit:delete_empty_adfiles = yes
        fruit:nfs_aces = no
     
    H

    hestermofet

    Ars Legatus Legionis
    24,887
  • #4
    Well, that was dumb. Thank you everyone for the help, and great recommendations, but it turned out to be a busted 10GBase-T SFP+ module. I'll see if I can replace it, but I'd eventually go back to optical as I found it more stable.

    I'm going to put atime back on. I have a good 3-2-1 strategy in place for backups, but I'm super paranoid about silent corruption and bitrot after decades of using optical discs for backups, so data integrity is important to me.
     
    • Like
    Reactions: continuum
    malor

    malor

    Ars Legatus Legionis
    16,093
  • #5
    atime has nothing to do with data integrity. It means that every time you read a file, data gets written to the filesystem, which is putting more load on your storage media for a non-useful reason. It increases your chance of data loss, rather than decreasing it.

    If bit rot is a worry, ZFS is one of the better solutions.

    And yes, fiber is just better than copper for anything over a gigabit. Among other things, it runs a lot cooler. A 10gb copper link is burning 10 watts per end for as long as you have the machine on, which adds up. It also gives you heat problems in switches,. and makes the electronics more likely to fail.

    EBay is festooned with Brocade SR multimode-fiber receivers for about $8 each. They seem to work very nicely.
     
    H

    hestermofet

    Ars Legatus Legionis
    24,887
  • #6
    So this is kind of shitty... turns out it's a bug in SMB compression for Windows 11 that was introduced in 22H2, and hasn't really been fixed yet entirely for all users.

    techcommunity.microsoft.com

    Slower SMB read performance for large files in 22H2

    Info and workaround for an issue seen in 22H2 Win11
    techcommunity.microsoft.com

    It's a regression introduced by a new feature that is supposed to speed up the copy of small files, but it unfortunately really slows down transfers of large files. I usually transfer... large files. Plus, it becomes especially apparant in benchmarking tools like CrystalDiskMark because they work by writing a large file to the "disk" being benchmarked, and then timing reading/writing it. My OS is fully patched, so whatever fixes MS has done in the last year aren't really helping me.

    I changed some of the infrastructure so I'm using 2.5GbE. I get a nice and healthy 296MB/s writes, but maxes out at 115MB/s reads, which tracks with the type of performance degradation other users are seeing. Moving to 2.5GbE and then still seeing almost exactly half the read speeds compared to writes really drove home it's not an issue with the TrueNAS box, it's some sort of issue with SMB or the network gear.

    I checked and checked and checked and it seems like the RJ45 SFP+ module I was using was generally fine. I knew I wasn't going crazy, and my earlier troubleshooting was sound, lol. I generally don't like 10GbE over RJ45 copper, and I used to use fiber. For all the reasons you mentioned and more, optical just works way better. The thing is, most of my high speed devices are upstairs while my network gear is downstairs. Until I can do some cable drops through the attic (something I've been meaning to do ever since I moved 5 years ago, but still haven't managed to do), I've been running cabling outside the house. You can find outdoor rated fiber optic, but it's still way more fragile than standard copper, so I have to change the cable almost yearly, after every winter. I changed to copper cabling this year, and it seems to work fine in terms of durability, but the winter just started, I guess. We'll see how it goes.
     
    Last edited:
    • Like
    Reactions: r0twhylr and Paladin
    malor

    malor

    Ars Legatus Legionis
    16,093
  • #7
    Maybe try armored fiber? It's got a metal sleeve that the fiber stays inside, and can survive being wallked on. I assume it's ice expansion that's messing up your runs, so the armor might help with that.

    Remember that paying a pro to do your fiber run through the attic is also an option.

    None of that will help with an SMB bug, though. Can you turn off compression on your TrueNAS box?

    If that doesn't help, remember Windows 10 LTSC is still an option. If you're running the IoT version, it'll get patches through 2032. But you'll probably have to pirate it. It's extremely difficult to buy a legit copy. I've actively tried.
     
    You must log in or register to reply here.
    Jump to latest Follow Reply
    Jump to latest Follow Reply