Agreed. When I was fresh out of university, my first job had me debugging embedded firmware for a device which had both a PowerPC processor as well as an ARM coprocessor. I remember many evenings staring at disassembled instructions in objdump, as well as getting good at endian conversions. This PPC processor was in big-endian and the ARM was little-endian, which is typical for those processor families. We did briefly consider synthesizing one of them to match the other's endianness, but this was deemed to be even more confusing haha
Your primary issue is going to be the power draw. If your electricity supplier has cheap rates, or if you have an abundance of solar power, then it could maybe find life as some sort of traffic analyzer or honeypot.
But I think even finding a PCI NIC nowadays will be rather difficult. And that CPU probably doesn't have any sort of virtualization extensions to make it competitive against, say, a Raspberry Pi 5.
To lay some foundation, a VLAN is akin to a separate network with separate Ethernet cables. That provides isolation between machines on different VLANs, but it also means each VLAN must be provisioned with routing, so as to reach destinations outside the VLAN.
Routers like OpenWRT often treat VLANs as if they were distinct NICs, so you can specify routing rules such that traffic to/from a VLAN can only be routed to WAN and nowhere else.
At a minimum, for an isolated VLAN that requires internet access, you would have to
define an IP subnet for your VLAN (ie /24 for IPv4 and /64 for IPv6)
advertise that subnet (DHCP for IPv4 and SLAAC for IPv6)
route the subnets to your WAN (NAT for IPv4; ideally no NAT66 for IPv6)
I quickly looked up the HPE/Aruba transceiver document, and starting on page 61 is the table of SFP+ transceivers, specifically describing the frequency and mode. At least from their transceivers, J9151A, J9151E, JL749A, and JL783A would work for your single-mode, 1310 nm needs.
You will have to do additional research to find generic parts which are equivalent to those transceivers. Good luck in your endeavors!
In some ways, I kinda despise the 802.3bz specification for 2.5 and 5 Gbps on twisted pair. It came into existence after 10 Gbps twisted-pair was standardized, and IMO exists only as a reaction to the stubbornly high price of 10 Gbps ports and the lack of adoption -- 1000 Mbps has been a mainstay and is often more than sufficient.
802.3bz is only defined for twisted pair and not fibre. So there aren't too many xcvrs that support it, and even fewer SFP+ ports will accept such xcvrs. As a result, the cheap route of buying an SFP+ card and a compatible xcvr is essentially off-the-table.
The only 802.3bz compatible PCIe card I've ever personally used is an Aquantia AQN-107 that I bought on sale in 2017. It has excellent support in Linux, and did do 10 Gbps line rate by my testing.
That said, I can't imagine that cards that do only 2.5 Gbps would somehow be less performant. 2.5 Gbps hardware is finding its way into gaming motherboards, so I would think the chips are mature enough that you can just buy any NIC and expect it to work, just like buying a 1000 Mbps NIC.
BTW, some of these 802.3bz NICs will eschew 10/100 Mbps support, because of the complexity of retaining that backwards compatibility. This is almost inconsequential in 2024, but I thought I'd mention it.
I've only looked briefly into APC/UPC adapters, although my intention was to do the opposite of your scenario. In my case, I already had LC/UPC terminated duplex fibre through the house, and I want to use it to move my ISP's ONT closer to my networking closet. That requires me to convert the ISP's SC/APC to LC/UPC at the current terminus, then convert it back in my wiring closet. I hadn't gotten past the planning stage for that move, though.
Although your ISP was kind enough to run this fibre for you, the price of 30 meters LC/UPC terminated fibre isn't terribly excessive (at least here in USA), so would it be possible to use their fibre as a pull-string to run new fibre instead? That would avoid all the adapters, although you'd have to be handy and careful with the pull forces allowed on a fibre.
But I digress. On the xcvr choice, I don't have any recommendations, as I'm on mobile. But one avenue is to look at a reputable switch manufacturer and find their xcvr list. The big manufacturers (Cisco, HPE/Aruba, etc) will have detailed spec sheets, so you can find the branded one that works for you. And then you can cross-reference that to cheaper, generic, compatible xcvrs.
In my first draft of an answer, I thought about mentioning GPON but then forgot. But now that you mention it, can you describe if the fibres they installed are terminated individually, or are paired up?
GPON uses just a single fibre for an entire neighborhood, whereas connectivity between servers uses two fibres, which are paired together as a single cable. The exception is for "bidirectional" xcvrs, which like GPON use just one fibre, but these are more of a stopgap than something voluntarily chosen.
Fortunately, two separate fibres can be paired together to operate as if they were part of the same cable; this is exactly why the LC and SC connectors come in a duplex (aka side-by-side) format.
Whereas in data center and networking, I have never seen anything but UPC, and that's what xcvrs will expect, with tiny exceptions or if they're GPON xcvrs.
So I need to correct my previous statement: to be fully functional as designed, the fiber and xcvr must match all of: wavelength, mode, connector, and the connector's polish.
The good news is that this should mostly be moot for your 30 meter run, since the extra losses from mismatched polish should still link up.
As for that xcvr, please note that it's an LRM, or Long Range Multimode xcvr. Would it probably work at 30 meters? Probably. But an LR xcvr that is single mode 1310 nm would be ideal.
Regarding future proofing, I would say that anyone laying single pairs of fibres is already going to constrain themselves when looking to the future. Take 100 Gbps xcvrs as an example: some use just the single pair (2 fibres total) to do 100 Gbps, but others use four pairs (8 fibres total) driving each at just 25 Gbps.
The latter are invariably cheaper to build, because 25 Gbps has been around for a while now; they're just shoving four optical paths into one xcvr module. But 100 Gbps on a single fiber pair? That's going to need something like DWDM which is both expensive and runs into fibre bandwidth limitations, since a single mode fibre is only single-mode for a given wavelength range.
So unless the single pair of fibre is the highest class that money can buy, cost and technical considerations may still make multiple multimode fibre cables a justifiable future-looking option. Multiplying fibres in a cable is likely to remain cheaper than advancing the state of laser optics in severely constrained form factors.
Naturally, a multiple single-mode cable would be even more future proofed, but at that point, just install conduit and be forever-proofed.
Starting with brass tacks, the way I'm reading the background info, your ISP was running fibre to your property, and while they were there, you asked them to run an additional, customer-owned fibre segment from your router (where the ISP's fibre has landed) to your server further inside the property. Both the ISP segment and this interior segment of fibre are identical single-mode fibres. The interior fibre segment is 30 meters.
Do I have that right? If so, my advice would be to identify the wavelength of that fibre, which can be found printed on the outer jacket. Do not rely on just the color of the jacket, and do not rely on whatever connector is terminating the fibre. The printed label is the final authority.
With the fibre's wavelength, you can then search online for transceivers (xcvrs) that match that wavelength and the connector type. Common connectors in a data center include LC duplex (very common), SC duplex (older), and MPO (newer). 1310 and 1550 nm are common single mode wavelengths, and 850 and 1300 nm are common multimode wavelengths. But other numbers are used; again, do not rely solely on jacket color. Any connector can terminate any mode of fibre, so you can't draw any conclusions there.
For the xcvr to operate reliably and within its design specs, you must match the mode, wavelength, and connector (and its polish). However, in a homelab, you can sometimes still establish link with mismatching fibres, but YMMV. And that practice would be totally unacceptable in a commercial or professional environment.
Ultimately, it boils down to link losses, which are high if there's a mismatch. But for really short distances, the xcvrs may still have enough power budget to make it work. Still, this is not using the device as intended, so you can't blame them if it one day stops working. As an aside, some xcvrs prescribe a minimum fibre distance, to prevent blowing up the receiver on the other end. But this really only shows up on extended distance, single mode xcvrs, on the order of 40 km or more.
Finally, multimode is not dead. Sure, many people believe it should be deprecated for greenfield applications. I agree. But I have also purchased multimode fibre for my homelab, precisely because I have an obscene number of SFP+ multimode, LC transceivers. The equivalent single mode xcvrs would cost more than $free so I just don't. Even better, these older xcvrs that I have are all genuine name-brand, pulled from actual service. Trying to debug fibre issues is a pain, so having a known quantity is a relief, even if it means my fibre is "outdated" but serviceable.
Fully agree. Sometimes the best equipment is that which is in-hand and thus free.
you can just send vlan tagged traffic across a dumb switch no problem
A small word of caution: some cheap unmanaged switches rigidly enforce 1500 Byte payload sizes, and if the switch has no clue that 802.1q VLAN tags even exist, will consider the extra 4 bytes as part of the payload. So your workable MTU for tagged traffic could now be 1496 Bytes.
Most traffic will likely traverse that switch just fine, but max-sized 1500 Byte payload frames with a VLAN tag may be dropped or cause checksum errors. Large file transfers tend to use the full MTU, so be aware of this if you see strange issues specific to tagged traffic.
Since you mentioned that your ONT is 2.5 Gbps, I am assuming that you need a twisted-pair NIC. I don't have a recommendation for a NIC exactly for 2.5 Gbps, but since you're specifically looking for low operating temperature, you may want to avoid 10 Gbps twisted-pair NICs.
10GBaseT -- sometimes called 10G copper, but 10Gbps DACs also use copper -- operates very hot, whether in an SFP+ module or as a NIC. The latter is observable just by looking at the relatively large heat sinks needed for some cards. This is an inevitable result of trying to push 800 MSymbols/sec over pairs of copper wires, and it's lucky to exceed 55 meters on CAT6. It's impressive how far copper wire has come, but the end is nigh.
Now, it could be that when a 10 Gbps NIC is only linked at 2.5 Gbps, it could drop into a lower power state. But my experience with the 10/100/1000 baseT specs suggest that the PHY on a 10 Gbps NIC will just repeat the signals four times, to produce the same transmission of the quarter-as-fast 2.5 Gbps spec. So possibly no heat savings there.
A dedicated 2.5 Gbps card would likely operate cooler and is more likely to be available as a single port, which would fit in your available PCIe ports. Whereas 802.3bz 2.5/5/10 Gbps NICs tend to be dual-port.
A final note: you might find "2.5 Gbps RJ45 SFP+" modules online. But I'm not aware of a formal 802.3 spec that defines the 2.5/5 Gbps speeds for modular connectors, so these modules probably won't work with SFP+ NICs.
Jk. But really, anything which helps organize stuff is a worthwhile job for a 3d printer. Even something to loop fibre optic cables on, so that they don’t exceed their maximum bend radii, is useful.
I think you’ll also find the 3d printer aids in other endeavors. I’ve used mine to print replacement car trims, ham radio accessories, a photo film spooler, a bushing to convert vacuum hose diameters, and other odds and ends.
I only have experience with Mellanox CX-5 100Gb cards at work, but my understanding is that mainline Linux has good support for the entire CX lineup. That said, newer kernel versions – starting at maybe 5.4? – will have all sorts of bug fixes, so hopefully your preferred distro has built with those driver modules included, or loadable.
As for Infiniband (IB), I think you’d need transceivers with specific support for IB. That Ethernet and IB share the (Q)SFP(+) modular connector does not guarantee compatibility, although a quick web search shows a number of transceivers and DACs that explicitly list support for both.
That said, are you interested in IB fabrics or what they can enable? One use-case native to IB is RDMA, but has since been brought to – so called “Converged” – Ethernet in the form of RoCE, in support of high-performance storage technologies like SPDK that enable things like NVMe storage over the network.
If all you’re looking for are the semantics of IB, and you’re only ever going to have two nodes that are direct-attached, then the Linux fabric abstractions can be used the same way you’d use IB. The debate of Converged Ethernet (CE) vs IB is more about whether/how CE switches can uphold the same guarantees that an IB fabric would. Direct attachment avoids these concerns outright.
So I think perhaps you can get normal 40 Gb Ethernet DACs to go with these, and still have the ability to play with fabric abstractions atop Ethernet (or IP if you use RoCE v2, but that’s not available on the CX-3).
Just bear in mind that IB and fabrics in general will get complicated very quickly, because they’re meant to support cluster or converged computing, which try to make compute and storage resources uniformly accessible. So while you can use fabrics to transport a whole NVMe namespace from a NAS to a client machine with near line-rate performance, or set up some incredible RPC bindings between two machines, there may be a large learning curve to achieve these.