SMB alternatives?

[redleader]

I've been experimenting with workstation network attached storage for a while now and never really found something I was entirely happy with. I have a synology system networked to a few workstation and development PCs over QSFP DACs, but SMB overhead makes that quite slow for large datasets and horribly bottlenecks anything NVMe. iSCSI would be faster, but I'd really like to have many to many networking since I've got a few workstations and development PCs. SMB Direct (rDMA) is really fast, but it forces Windows Server on the NAS, which I'm not really excited about for management/hosting reasons and of course it doesn't support regular Windows 11 Home/Pro at all.
Anything else I should be looking at?

 

[continuum]

We're deploying mostly RoCE (RDMA) and NFSv4 lately over QSFP DAC, but it sounds like that doesn't help you.

 

[sedawkgrep]

Copyparty was posted the other day and it really, really looks amazing. There's a video demonstration of all it can do.

To top it off, it's all self-contained.

https://github.com/9001/copyparty

It may not meet your speed requirements, but it certainly is incredibly flexible.


View: https://www.youtube.com/watch?v=15_-hgsX2V0

 

[Ardax]

Off the top of my head, I can't think of anything that isn't an uptick in complexity.

RDNA support in Samba is still in prototype stage. If NFS has anything similar, I'd be surprised if Windows NFS client supported it. Maybe there's alternatives there?

Or you do iSCSI and some kind of clustered file system?

If you really need the performance that badly, "install Windows Server" is probably the path of least resistance. At least OpenSSH built in and WSL make for a reasonably good way to run Linux stuff if you still need it.

Also oranging this thread for further replies.

Copyparty was posted the other day and it really, really looks amazing.

That looks pretty cool, but the warnings in the SMB section are pretty dire.

 

[gusgizmo]

Have you done any tuning to SMB or your network? Not sure the extent that is possible but I assume you have logging turned down or off, and the typical accelerators like sendfile and async setup? Jumbo frames? Again apologize if this stuff is obvious but it's good to get a baseline.

Synology sort of implies a lower end CPU. That can be limiting for SMB as it's 1 file = 1 thread.

If windows server is out of the question, is a custom linux build also a no-go?

How about windows server on your workstations and divorce yourself from bing and copilot? Then you can do starwind or csv on-top of iscsi?

 

[redleader]

I have a Windows Server setup now for testing, and I can get >3GB/s real-world through it using RDMA which is awesome, but overall its an annoying solution since I'd have to upgrade every the OS on every client since no support on Home/Pro. Likewise while my impression is that storage spaces or whatever its called is basically fine, I'm a little hesitant about building out a big RAID on Windows, especially since I've never done data recovery after a dead disk on it before.

 

[Ardax]

since I'd have to upgrade every the OS on every client since no support on Home/Pro.

Hold up. What? You're telling me this doesn't work on Win 11 Pro?

 

[redleader]

Hold up. What? You're telling me this doesn't work on Win 11 Pro?

View attachment 115836

Yeah, there's a special version of Windows that enables rdma:

https://meincmagazine.com/civis/threads/rdma-on-windows.1493291/#post-42038804

Edit: This suggests that Pro can if you use a new enough install media. I couldn't get it to work on a clean install of Pro in summer 2023, but maybe that wasn't available yet?

 

[Ardax]

Microsoft says you need at least Windows 10 22H2. https://learn.microsoft.com/en-us/windows-server/storage/file-server/smb-direct

You need an appropriate NIC too, as explained in the other thread. (That I also participated in. LOL)

 

[drnick1]

Anything else I should be looking at?

NFS on Linux. Recent Windows versions come with native client support, IIRC.

 

[gusgizmo]

I have a Windows Server setup now for testing, and I can get >3GB/s real-world through it using RDMA which is awesome, but overall its an annoying solution since I'd have to upgrade every the OS on every client since no support on Home/Pro. Likewise while my impression is that storage spaces or whatever its called is basically fine, I'm a little hesitant about building out a big RAID on Windows, especially since I've never done data recovery after a dead disk on it before.

Whats the underlying storage? SATA/SAS, SSD or Spinning Rust? NVME?

Storage spaces is solid. I wouldn't be too worried about recovery. I've even been able to mount volumes from windows pro. Making it perform is a little harder. You can turn off synchronous writes but I'm not a fan of that. There is caching available but it's not quite like ZIL.

I know I'm somewhat of an outlier here, but LSI makes great adapters for cheap enough. Put a BBU on the card, flip the switch for write caching, you safely have 95% of the peak performance of that array and you can put any OS and filesystem on top.