2.5 GE, more speed, more headaches

[BigLan]

2.5GbE and 5GbE are explicitly intended to be drop-in upgrades for homes and small businesses that won't require a cabling upgrade, both intended to work on Cat5e. (5Gb does need Cat6 for longer distances, but not Cat6a like 10Gb, and many places future-proofed a bit and already have Cat6 if they've had wiring installed in the last several years so for many users it is still a drop-in upgrade.)

It was more the stapling and cable routing done by the builder, coupled with the shoddy job I did with cheap keystones that surprised me it just worked. I couldn't remember if it was 5 or 5e cabling either, and it's a longer run - probably > 100 feet as it snaked through walls and the attic.

I wasn't that worried about the newer runs I'd put in to the media server and game room PCs (cheap keystones aside.)

Also had to use a USB connection for one PC which has a vertical GPU and it seemed to survive reboots and sleep states pretty well so far.

 

[CatBus]

stapling

I know precisely what you're talking about...

 

[CatBus]

10GBASE-T allows for Cat6 for runs up to 55m by spec. In a home environment, Cat6 should be fine for 10Gbps. 55m is a long run, even if you have to go weird routes to get there, and that's just "spec." You can probably get away with even longer runs in a relatively quiet home environment where you have a few runs, not 50-100 (I recognize the amount of Cat6 in my house is very, very atypical).

As long as you're talking out-of-spec performance, Cat5e can theoretically get you 10GBase-T up to ~45m, which is also a long run for a house. May as well test your old cables before you go through the trouble of tearing them out.

 

[ERIFNOMI]

As long as you're talking out-of-spec performance, Cat5e can theoretically get you 10GBase-T up to ~45m, which is also a long run for a house. May as well test your old cables before you go through the trouble of tearing them out.

I'm talking in-spec. The spec says 55m Cat6. You need Cat6A for the full 100m.

The need to Cat6A isn't for the increased bandwidth (250MHz vs 500MHz) but the reduced crosstalk. A shorter run also suffers less crosstalk. You also get less crosstalk by having fewer parallel runs (alien crosstalk).

 

[torp]

This is going to sound stupid, but it's only a bottleneck if it's a bottleneck. [snip] you can drop those on the second switch.

Yeah thanks everyone, you got me thinking. New plan is to get an 8 port 2.5 Gbe as that would be plenty for the devices that really need >= 1 Gpbs once in a while, and at the moment I have at most 3 that can use the 2.5 speed. I can probably even match it with an older 8 port 1 Gbps that's gathering dust in a box instead of the 16 port...

Your use case doesn't sound too demanding, [snip]

A slightly more upscale variation on the same theme is to pick up two of something like this:
https://www.servethehome.com/gigaplus-gp-s25-0802-8-port-2-5gbe-and-2-port-10g-review/

There's an even nicer article linked in the review you posted:

https://www.servethehome.com/the-ul...gear-hasivo-mokerlink-trendnet-zyxel-tp-link/

Shopping time

 

[iljitsch]

It did turn out I still have a few cat 5 (no e) cables and those won't do 2.5 G. I used cat 6a between the living room and my home office, so that should be good for 10 G whenever that lands.

 

[Demento]

It did turn out I still have a few cat 5 (no e) cables and those won't do 2.5 G. I used cat 6a between the living room and my home office, so that should be good for 10 G whenever that lands.

That's quite surprising. I've had the junkiest cables do 2.5gb so long as it's a short run. 2.5/5Gb pretty much exists for using cables that can't do 10 gig.

 

[iljitsch]

It's a 5 meter cable I think, with factory-attached plugs so I was also somewhat surprised. Still, this cable could be 25 years old. It has the code E190560 on it, no idea what that means.

 

[RicDavis]

I've seen some old "Category 5" cables that only have two pairs in them rather than four. Work fine at 100 Mbit, obviously less useful at gigabit. If only I hadn't encountered some of them in the structured cabling of an entire building.....

 

[iljitsch]

It says "4PR" on it so that doesn't apply here... it also does 1 G just fine.

I'm trying to remember when I first got UTP cables... I got a PCMCIA NIC for my Amiga 1200 in 1995 and then not too long afterwards an NE2000 clone for my Amiga 3000, but I probably used coax between those at first. But for sure I got UTP in 2000 when I gave up on the Amiga coming back to life and bought some PC hardware with blazing fast 3Com 10/100 NICs.

And why ever throw away a perfectly good UTP cable, so I'm sure I still have a few that are now a quarter century old.

 

[Num Lock]

Just because a port is up at 1G or 2.5G doesn't mean the device can actually pass that traffic at line rate. I'd suggest putting money towards a better quality product vs getting more big ports for less. The better the apparent value, the more likely that switch is cutting more corners on the inside. If you've got a bottleneck at home with 1G ports, something is off.

 

[ERIFNOMI]

Just because a port is up at 1G or 2.5G doesn't mean the device can actually pass that traffic at line rate. I'd suggest putting money towards a better quality product vs getting more big ports for less. The better the apparent value, the more likely that switch is cutting more corners on the inside. If you've got a bottleneck at home with 1G ports, something is off.

1 and 2.5Gbps switches are nothing special. You'd be hard pressed to find a small switch that can't switch at line rate on all ports. Servethehome has been buying the shittiest no-name 2.5Gbps switches they can find from the darkest hellholes of the internet and I don't think they've found one that can't keep up yet.

For a dumb switch that just has to switch, there's really nothing to fuck up. The switch ASIC comes off the shelf, someone else makes the PSU, all they do is print a PCB and wire up jellybean parts.

On the other side of things, it's extremely easy to saturate 1Gbps. Like stupid easy. A half decent HDD can saturate with sequential reads or writes. 1Gbps is dog slow for even a shit SSD.

 

[iljitsch]

I tested that red cable again as well as an old yellow cable (one that came with a Cisco router long ago) and now both cat 5 cables apparently do 2.5 G...

 

[CatBus]

1 and 2.5Gbps switches are nothing special. You'd be hard pressed to find a small switch that can't switch at line rate on all ports. Servethehome has been buying the shittiest no-name 2.5Gbps switches they can find from the darkest hellholes of the internet and I don't think they've found one that can't keep up yet.

Yeah, the only measurable speed difference you get between cheap crap and nicer gear is in the aggregate transfer speeds when you're saturating ALL ports simultaneously, and even then, it's a pretty subtle difference, and not one a typical home user would notice or care about.

shittiest no-name 2.5Gbps switches they can find from the darkest hellholes of the internet

LOL, this is so, so true. And yet, I've bought three of them already and they're all quite good enough for my "it's just my house, not NORAD" purposes.

 

[BigLan]

Yeah, the only measurable speed difference you get between cheap crap and nicer gear is in the aggregate transfer speeds when you're saturating ALL ports simultaneously, and even then, it's a pretty subtle difference, and not one a typical home user would notice or care about.



LOL, this is so, so true. And yet, I've bought three of them already and they're all quite good enough for my "it's just my house, not NORAD" purposes.

Yup, my "Hisource" ali special has met my expectations so far. I'm pretty sure they're all just cloning the basic design by now and sticking whatever name they feel like on the enclosure. The quick reviews of them I saw on ServeTheHome looked like they can do >39gbps out of a theoretical 40gbps which they noted, but isn't going to make a difference to me copying Steam games between PCs.

Dumb networking question: is it worth enabling jumbo frames on the 2.5gb cards? and is 4k, 9k or 16k better? (I suspect the answer will be "you won't notice the difference anyway")

 

[ERIFNOMI]

Yup, my "Hisource" ali special has met my expectations so far. I'm pretty sure they're all just cloning the basic design by now and sticking whatever name they feel like on the enclosure. The quick reviews of them I saw on ServeTheHome looked like they can do >39gbps out of a theoretical 40gbps which they noted, but isn't going to make a difference to me copying Steam games between PCs.

Dumb networking question: is it worth enabling jumbo frames on the 2.5gb cards? and is 4k, 9k or 16k better? (I suspect the answer will be "you won't notice the difference anyway")

I do 10Gbps without jumbo frames just fine.

 

[CatBus]

Dumb networking question: is it worth enabling jumbo frames on the 2.5gb cards? and is 4k, 9k or 16k better? (I suspect the answer will be "you won't notice the difference anyway")

Not in my experience. IMO it'd be more likely to create problems than solve them.

 

[iljitsch]

About jumbo frames: it depends on whether the NIC supports TSO. That's TCP segmentation offloading, where the OS gives a super large packet to the NIC and the NIC splits it up in smaller packets. And probably the reverse when receiving packets, but I'm not sure. With that in place, jumboframes don't buy you anything. But without TSO, you at least get better CPU usage and maybe also better performance.

My experience has been that it doesn't matter for IPv4, but it's helpful for IPv6. And also that changing stuff like VLANs and MTUs on the Mac has gotten way worse the past decade.

 

[Hap]

50-100 (I recognize the amount of Cat6 in my house is very, very atypical).

If there are two of us in the same thread, I would say drop one of the "very"

 

[iljitsch]

Is there any advantage in going for cat 6 over cat 6a? I did a decent run of 6a through two rooms, four closets and a hallway, and the stuff is pretty thick and pretty stiff. But I like the idea that apart from probably not having done the connectors to spec, this is the ultimate in (U)TP (it's not unshielded...) and the next step would be fiber.

 

[steelghost]

6 is cheaper and easier to run. The only real benefit I can think of for 6a over 6 is that it's "in spec" for 10GBaseT, and that's not something you generally want unless you have no other choice, eg I have one system that uses it because it has 10GBaseT onboard and all the PCIe slots are occupied with other things. Fortunately it's using a 1m pre-made 6a patch lead in a rack, so I don't need to worry about the termination.

If you actually want / need 10G throughput, then you're best off running fibre (as I'm sure you're aware), being as it is neither thick nor stiff, and doesn't care about EM.

 

[continuum]

6 is significantly easier to run than 6a. But if you've dealt with it before and found it fine then have at it.

 

[cerberusTI]

On switches, I bought a cheap, fanless 8 10/100/1000/2500 port and 2 SFP+ port switch for under $70 a while ago and it works great. A bad switch can be frustrating, but 2.5g is pretty cheap these days, and they do not seem problematic at all.

On cables, better cables can matter. I became tired of intermittent issues like a decade ago and ran shielded cable. Any packet loss even in a very high noise environment just disappeared and everything always now does basically max speed. You need preterminated cables or special equipment and metal plugs though, and if you have more than one ground you must be very careful of how it is grounded.

If you want 10g or more, and it is not built in already, you want fiber. Get an SFP+ switch and run that instead.

 

[iljitsch]

6 is significantly easier to run than 6a. But if you've dealt with it before and found it fine then have at it.

Well, the hard part of this run was drilling holes in a bunch of brick walls, the cable itself didn't give me major trouble, although something more bendy would have been a bit easier.

The downside of fiber is that you need fiber-capable stuff on both ends. With cat 6a I have a seamless upgrade path from 1 to 2.5 to 5 to 10 G.

 

[Lord Evermore]

it's "in spec" for 10GBaseT, and that's not something you generally want unless you have no other choice, eg I have one system that uses it because it has 10GBaseT onboard and all the PCIe slots are occupied with other things

I don't fully understand this statement. You DON'T want the ability to run 10GbE? I get it if you meant it's a chipset that doesn't support 2.5 or 5, just 1 and 10, and you don't have a 10GbE switch so you'd be stuck at 1Gb, but other than that? (And you're using that Cat6a patch cable so I assume you DO have a 10Gb switch.) And regardless of what your current hardware supports, why would you NOT be at least considering still having the copper be capable of 10Gb in the future?

 

[ERIFNOMI]

I don't fully understand this statement. You DON'T want the ability to run 10GbE? I get it if you meant it's a chipset that doesn't support 2.5 or 5, just 1 and 10, and you don't have a 10GbE switch so you'd be stuck at 1Gb, but other than that? (And you're using that Cat6a patch cable so I assume you DO have a 10Gb switch.) And regardless of what your current hardware supports, why would you NOT be at least considering still having the copper be capable of 10Gb in the future?

If you hadn't trimmed their comment in your quote, you'd have the rest of the context to understand what they were saying.

They suggest going fiber once you go to 10Gbps. You can agree or disagree with that, but that's obviously what they were saying, not that you'd never want 10Gbps.

 

[Lord Evermore]

If you hadn't trimmed their comment in your quote, you'd have the rest of the context to understand what they were saying.

I trimmed it because it seemed irrelevant. The first and second paragraphs were not explicitly linked in my mind. A literal reading of the sentence says you don't want Cat 6a cabling that is capable of 10GbE instead of Cat 6 unless you have no choice, which makes no sense. (Don't hold it against me that I read things as they're written and then ASK for clarification.) Just because fiber is a good way of doing 10Gb doesn't automatically mean Cat 6a is a bad choice. @iljitsch pointed that out in the post immediately afterward.

 

[cerberusTI]

I trimmed it because it seemed irrelevant. The first and second paragraphs were not explicitly linked in my mind. A literal reading of the sentence says you don't want Cat 6a cabling that is capable of 10GbE instead of Cat 6 unless you have no choice, which makes no sense. (Don't hold it against me that I read things as they're written and then ASK for clarification.) Just because fiber is a good way of doing 10Gb doesn't automatically mean Cat 6a is a bad choice. @iljitsch pointed that out in the post immediately afterward.

I think the point is more that copper does not do this speed easily at this time. The switch will be expensive, will likely have enough power use to require more careful placement (and an internal fan), and you will need to ensure it is all correctly grounded to a single point.

Fiber is cheaper and easier at that speed, unless the device already has a 10gb copper port (which is where you may lack a choice.)

 

[steelghost]

Oh man, I leave this thread for a few days....

In my post I very specifically and deliberately used the term "10GBaseT", not just "10 gig ethernet" or similar, because I was answering @iljitsch's question...

Is there any advantage in going for cat 6 over cat 6a?

...in part by flipping it, ie "is there any good reason to go for 6a?" And the one reason that you might use 6a is if you wanted to get (or have the option of) 10 gigabit throughput, and you had reasons to want to do that over copper eg you wanted to be able to use your existing gigabit / multigig gear in the meantime.

If you are running a new link over any significant distance with the nearterm goal of 10 gigabit or better throughput, you are probably better off going with fibre if you have a choice. I elaborate on my personal reasons for this view below, but in short, 10GBaseT is probably too energy hungry to be tolerable for most home users, especially when fibre is as accessible and relatively inexpensive as it has become.

Spoiler: Hidden because longish and not directly relevant

In my specific case, I recently completed a small network rack update. The upper server in the rack is built around the snappily named ASRock Rack E3C246D4U2-2T, which has two onboard 10GBaseT ethernet ports. I have two switches in use, each with two SFP+ ports.

I actually bought a passively cooled 10GBaseT switch with the goal of having a single uplink from the 10G switch to my main 24-port switch, and then connecting both 10G ports of my main server, plus a single link to my cold backup box, to the remaining three 10G ports.

I abandoned this approach because the aggregate heat output (and accompanying electricity consumption) was just too much. So, I reconfigured my server so that the TrueNAS VM uses a virtual NIC connected to the same bridge as the Proxmox server. That means I only need a single 10G port for that server, and by accepting a slower link for the cold backup server, well, I save myself around 30W of constant power draw.

(I will say though, if you find yourself in need of a switch that has x4 10GBaseT ports without the use of expensive and hot SFP+ modules, the CRS304-4XG-IN is a rather nice bit of kit. I should sell it really, but I can't help feeling I'd regret it down the road....)

So, as things stand I have two live 10GBaseT ports, one on the main server, one in an SFP+ module on my switch, linked by a single Cat6a patch cable. The 10G uplink between my two switches is a short DAC cable, and when I get around to running it, the fast link to my office switch will be armoured OM4 fibre. In my office my main PC has an X520 card which is connected to my office switch with another short DAC, which is a reasonably low power solution all round.

 

[cerberusTI]

Oh man, I leave this thread for a few days....

In my post I very specifically and deliberately used the term "10GBaseT", not just "10 gig ethernet" or similar, because I was answering @iljitsch's question...

...in part by flipping it, ie "is there any good reason to go for 6a?" And the one reason that you might use 6a is if you wanted to get (or have the option of) 10 gigabit throughput, and you had reasons to want to do that over copper eg you wanted to be able to use your existing gigabit / multigig gear in the meantime.

If you are running a new link over any significant distance with the nearterm goal of 10 gigabit or better throughput, you are probably better off going with fibre if you have a choice. I elaborate on my personal reasons for this view below, but in short, 10GBaseT is probably too energy hungry to be tolerable for most home users, especially when fibre is as accessible and relatively inexpensive as it has become.

Spoiler: Hidden because longish and not directly relevant

In my specific case, I recently completed a small network rack update. The upper server in the rack is built around the snappily named ASRock Rack E3C246D4U2-2T, which has two onboard 10GBaseT ethernet ports. I have two switches in use, each with two SFP+ ports.

I actually bought a passively cooled 10GBaseT switch with the goal of having a single uplink from the 10G switch to my main 24-port switch, and then connecting both 10G ports of my main server, plus a single link to my cold backup box, to the remaining three 10G ports.

I abandoned this approach because the aggregate heat output (and accompanying electricity consumption) was just too much. So, I reconfigured my server so that the TrueNAS VM uses a virtual NIC connected to the same bridge as the Proxmox server. That means I only need a single 10G port for that server, and by accepting a slower link for the cold backup server, well, I save myself around 30W of constant power draw.

(I will say though, if you find yourself in need of a switch that has x4 10GBaseT ports without the use of expensive and hot SFP+ modules, the CRS304-4XG-IN is a rather nice bit of kit. I should sell it really, but I can't help feeling I'd regret it down the road....)

So, as things stand I have two live 10GBaseT ports, one on the main server, one in an SFP+ module on my switch, linked by a single Cat6a patch cable. The 10G uplink between my two switches is a short DAC cable, and when I get around to running it, the fast link to my office switch will be armoured OM4 fibre. In my office my main PC has an X520 card which is connected to my office switch with another short DAC, which is a reasonably low power solution all round.

I looked at 10gb switches, but they mostly use so much power they need a fan, and the ones which do not come with one use enough power that they probably should have one. It seemed impractical, even if possible.

I like to drop my network equipment into a deep hot hole between the power equipment (mostly as it keeps the cats from being able to reach the network cords, regardless of how much they reach). The less than 2 watts my 8 port 2.5 2 port 10g spf switch mostly uses, or the 5 watts or so it uses under as much load as I could arrange, is still easily dissipated without any real airflow while remaining well under its temperature limit.

That is an interesting switch, and is much lower power than others I have seen, but is still enough to cause practical problems when used in the home, as you found. I would use 6a (or in practice almost certainly cat 8) anyway if installing new wire unless the cost is a huge issue (I would see the time to do that as way more bothersome), but I would run 2.5 over it for now.

If 10gb is the goal, fiber is better in enough ways to make it worth running, and few devices even have a 10gb port built in right now. Switches with an SFP+ port or two are cheap enough that placing one to handle the fiber link is not a big deal, unless the situation is a very odd one.

 

[BigLan]

Is there any talk about being able to use cat 5e or 6 for 10gb? I remember when 5e was only good for 1gb, but now it can handle 2.5 and is rated for 5.

 

[evan_s]

Is there any talk about being able to use cat 5e or 6 for 10gb? I remember when 5e was only good for 1gb, but now it can handle 2.5 and is rated for 5.

I doubt the spec will change.

category 5e or 6 may reach up to 55 metres (180 ft) depending on the quality of installation.

https://en.wikipedia.org/wiki/10_Gigabit_Ethernet#10GBASE-T

The line encoding used by 10GBASE-T is the basis for the newer and slower 2.5GBASE-T and 5GBASE-T standard, implementing a 2.5 or 5.0 Gbit/s connection over existing category 5e or 6 cabling.[51] Cables that will not function reliably with 10GBASE-T may successfully operate with 2.5GBASE-T or 5GBASE-T if supported by both ends.

https://en.wikipedia.org/wiki/2.5GBASE-T_and_5GBASE-T

2.5gb works because it's the 10gb signaling/encoding improvements without increasing the bandwidth for the signal so has the same requirements a 1gb Ethernet. 5gb increases the bandwidth while still keeping it low enough that it is in spec for Cat5e and 6 cables.

 

[steelghost]

View: https://youtu.be/_HaLU3ecUSY?si=8-TvWrtKEcgOeVB1


This video (and plenty of others from what I saw) seem to suggest you have a good chance of it working, at least for shorter runs. It's going to depend on the general quality of your cable plant of course. But as @evan_s says, the spec isn't going to change.

 

[ERIFNOMI]

Is there any talk about being able to use cat 5e or 6 for 10gb? I remember when 5e was only good for 1gb, but now it can handle 2.5 and is rated for 5.

Cat6 is spec for 10GBASE-T at a reduced length of 55m.

The entire point of 2.5GBASE-T and 5GBASE-T was to provide something in-between 1G and 10G that would work with existing cables because 10G (on twisted pair) mostly stalled out. They're pretty recent standards introduced long after cat5e, 1000BASE-T, and 10GBASE-T.

 

[CatBus]

I looked at 10gb switches, but they mostly use so much power they need a fan, and the ones which do not come with one use enough power that they probably should have one. It seemed impractical, even if possible.

FWIW on the cheap no-name Chinese gear front, I picked this up, and it's fanless and runs pretty cool/low power. I'd call it "barely managed" more than managed, but it more than meets my needs. Just don't put 10GBase-T modules in it, or that whole fanless business turns into a bad idea (there's a whole YouTube video about a guy who tries to load this model up with 10GBase-T modules, and... it goes as expected).

https://www.amazon.com/Managed-8X10G-SFP-Aggregation-Multi-gig/dp/B0CQJCQ17Q?th=1

 

[cerberusTI]

FWIW on the cheap no-name Chinese gear front, I picked this up, and it's fanless and runs pretty cool/low power. I'd call it "barely managed" more than managed, but it more than meets my needs. Just don't put 10GBase-T modules in it, or that whole fanless business turns into a bad idea (there's a whole YouTube video about a guy who tries to load this model up with 10GBase-T modules, and... it goes as expected).

https://www.amazon.com/Managed-8X10G-SFP-Aggregation-Multi-gig/dp/B0CQJCQ17Q?th=1

That is the point I suppose. If you start using 10GBase-T in any volume for home use it is probably a bad idea compared to alternatives.

It can maybe get more efficient in the future, but for now a switch like that using fiber is the way to go. It is also notable that the link you sent has a $35 option for 4 2.5g ports and 2 SFP+ 10g ports, which you can cheaply use for longer links.

 

[BigLan]

I doubt the spec will change.



https://en.wikipedia.org/wiki/10_Gigabit_Ethernet#10GBASE-T


https://en.wikipedia.org/wiki/2.5GBASE-T_and_5GBASE-T

2.5gb works because it's the 10gb signaling/encoding improvements without increasing the bandwidth for the signal so has the same requirements a 1gb Ethernet. 5gb increases the bandwidth while still keeping it low enough that it is in spec for Cat5e and 6 cables.

Thanks, and the others who chimed in. You tech nerds are why I love hanging out at Ars

 

[CatBus]

If you start using 10GBase-T in any volume for home use it is probably a bad idea compared to alternatives.

Oh definitely. But small, cool fanless 10G switches do exist, with the caveat that you use them properly. Properly in these cases usually means avoid 10GBase-T like it was kryptonite, which is just good advice in general.

 

[Lord Evermore]

Remember when dropping your exposed computer equipment (with no moving parts) into a vat of mineral oil to dissipate heat was the thing? I'd love to open an IT room door and see a large aquarium full of un-cased switches and servers soaking in mineral oil, which of course is being pumped out to large radiators somewhere else. Preferably rigged with a cinematically-awesome robotic system for extracting each unit for maintenance/upgrades or plugging in cabling. And maybe a walkway above it with no safety railing for when the sysadmin needs to have a duel with a beancounter.

 

[KD5MDK]

The entire point of 2.5GBASE-T and 5GBASE-T was to provide something in-between 1G and 10G that would work with existing cables because 10G (on twisted pair) mostly stalled out. They're pretty recent standards introduced long after cat5e, 1000BASE-T, and 10GBASE-T.

Also the ethernet port does not know what kind of cable is plugged into it, so if it is able to negotiate a link at whatever speed it will use that and not decide not to try because of the nominal cable rating.