- Jul 19, 2015
- 818
- 1,359
Rich (BB code):
▒▒▖ ▒▒▒▒▖ ▒▒▒▒▒▖ ▒▒▖▒▒▒▖ ▒▒▖ ▒▒▒▒▖ ▒▒▒▒▒▖ ▒▒▒▒▖ ▒▒▖ ▒▒▖
▒▒▌ ▒▒▛▀▒▒▖▒▒▛▀▒▒▖ ▒▒▌▒▒▒▒▖▒▒▌ ▒▒▛▀▒▒▖ ▒▒▛▀▒▒▖▒▒▛▀▒▒▖▝▒▒▒▒▛▘
▒▒▌ ▒▒▒▒▒▒▌▒▒▒▒▒▛▘▒▒▒▖▒▒▌▒▒▛▒▒▒▒▌▒▒▒▖▒▒▒▒▒▒▌▒▒▒▖▒▒▒▒▒▛▘▒▒▌ ▒▒▌ ▝▒▒▛▘
▒▒▌ ▒▒▛▀▒▒▌▒▒▛▀▒▒▖▝▀▀▘▒▒▌▒▒▌▝▒▒▒▌▝▀▀▘▒▒▛▀▒▒▌▝▀▀▘▒▒▛▀▒▒▖▒▒▌ ▒▒▌ ▒▒▒▒▖
▒▒▒▒▖▒▒▌ ▒▒▌▒▒▒▒▒▛▘ ▒▒▌▒▒▌ ▝▒▒▌ ▒▒▌ ▒▒▌ ▒▒▒▒▒▛▘▝▒▒▒▒▛▘▒▒▛▀▒▒▖
▝▀▀▀▘▝▀▘ ▝▀▘▝▀▀▀▀▘ ▝▀▘▝▀▘ ▝▀▘ ▝▀▘ ▝▀▘ ▝▀▀▀▀▘ ▝▀▀▀▘ ▝▀▘ ▝▀▘So now I have had some additional “free” time enforced upon me I figured it was time to write up my ongoing side project.
It all started when I happened across this web page. A Pi Zero with Ethernet & two USB-A ports, what’s not to love!? For the longest time though I had no need for one & shipping from the UK would probably be meh so I just bookmarked it & carried on with life. Somewhere later on down the track I got stuck into overhauling my network, it started with better wifi, then some fibre, better switches, 10Gb, bonding, etc. Along the way I had to learn more about some core networking technologies. Things like LACP, spanning tree and & its myraid variations, IGMP, LLDP, protocol snooping, etc. It’s been a journey that’s still far from being “done”¹ ².
Eventually breaking my home network over & over again started to get super old & how do I even know if the config & hardware I’m bolting together are even working as intended. I needed a test rig, it needed to be portable, easy to power, have a wired network interface (ideally more than one), some wireless interfaces would be nice to have as well & it needed to be easy (for me) to reconfigure or repurpose or nuke & pave or whatever I needed. This began when I wanted to know if the DHCP snooping I had spent far too long learning & deploying was, well, snooping on DHCP. So, perfect excuse, I pulled up that bookmarked RISC OS bits page and put in an order. A couple weeks or so later this landed on my doorstep.
I took a punt that whatever hardware they were bolting on to the Pi Zero to expand the I/O was going to be something normal enough to work with Linux. As I found out later on, this is very much the case so after taking 10 minutes to look nostalgically at RISC OS I dd’d a Linux image onto the MicroSD card, set to work configuring dhcpd & plugged it into just about every piece of networking hardware I could find to see if it was able to hand out leases or not. Using a combination of device logs, dhcpdump & nmap I got the answers I was looking for. Success! and validation of the idea that small, low power hardware would be a viable approach to labbing. This philosophy is somewhat stolen from MikroTik, with a few exceptions a $30 hAP mini will do most of the same things, using the same OS, as a $300 CRS 3 series switch or a $1,000 CCR router, usually just slower or with lower maximums or the like. Validating that I could apply the same style of approach to my own needs was super satisfying, my rouge DHCP server didn’t need to be able to handle 1,000 clients, it just needed to poke whatever it was connected too until it got a response.
So cool, now I have a Pi Zero with Ethernet & two USB-A ports, it’s useful, but it lacks any wireless capabilities & has a 40 pin GPIO header going unused. Mostly in parallel to rolling out DHCP snooping I had also been configuring RSTP across my whole network properly³. This set me thinking, as part of upgrading what I can to dual 10Gb connections using LACP⁴ I now have a bunch of hardware that has 1Gb onboard Ethernet ports going unused, if I can get my head around configuring RSTP on Linux then I could use those ports to provide redundant links between those devices and the 1Gb switch in my rack, should both of the links in the 10G LAGG go down or the 10G switch itself. A quick google later lead me to the Pi Jack. It’s basically an enc28j60 Ethernet - SPI bridge hooked up the the GPIO header on the Pi & a device tree overlay to configure the pins. It’s also been out of production since forever. No matter though, The enc28j60 is used everywhere and the clone boards that came after are arguably better, allowing for user config of slave selection using jumpers. A quick trip to Amazon (or Ebay or Aliexpress) and a spare hAP lite later & I had a dual ethernet Raspberry Pi Zero. Granted the enc28j60 board ran at all of 10Mb/s & the dual USB/ethernet board that connected using pogo pins to the test pads on the underside of the Pi would do 100Mb/s & used the same chipset as the Pi 2 to get there but no matter, I didn't need speed, I needed interfaces. For funsies there is a hAP mini on top of the fridge in the kitchen thats acting as the near side of an nstream bridge & the hAP lite as the far side. This is potentially the least efficient way to do this vs just using wifi, but meh, its more "fun".
Ultimately this ended up being a fantastic platform to learn the specifics of (R)STP on linux with my eventual solution being to use the MSTPd userspace daemon and NetworkManager-dispatcher to trigger a script to handle runtime configuration of the bridge as the interface came up. Somewhere around this time I also moved to a new ISP that provided native IPv6 and after learning a whole bunch more mind melting weirdness about that protocol, used this rig as my test bed to migrate from NetworkManager (using DHCPCd, dnsmasq, chrony, wpa_supplicant & NetworkManager-dispatcher) to systemd-networkd, systemd-timesyncd, systemd-resolved, iwd & networkd-dispatcher. At this point I figure I have gotten my moneys worth out of this board, using it as a test bed means migrating the rest of my network, the bits that actually matter, has gone substantially smoother.... But, it doesnt have WiFi, or Bluetooth, it doesn't need either buuuuuut I mean, it would be kind of cool. A quick trip to Core Electronics solved problem in short order.
For a while everything was good, but then I got to thinking, I have 2 more ethernet ports on that hAP and a spare Pi zero. A bit of google sleuthing around lead me to this Tindie page, Kind of looks alot like this is where RISC OS Bits are getting their boards from, maybe. Regardless, one order later & another one was in the post, thats the wired bit solved. Next up was the wireless, There have been a number of wifi boards back in the day for the Pi Zero, but I jumped on this hype train far too late to source any of the early ones (see here, here or here for a few examples). There was one name that kept coming up though, the now defunct Red Bear electronics and their IoT pHAT. The company may be gone but as long as there are still boards for sale & a git repo to clone it looks like a viable option. Hitting up The IoT Store saw one on my doorstep pretty quickly and a trip to ebay landed me some nicely color coded GPIO headers.
A little patience, some filthy soldering and a bit of luck later & my second lab rig was beginning to look like ready
Not much later The USB/ethernet board arrived and this rig was built
But with every problem solved, a new one is often created. This whole lab lives in the corner of the laundry room in the back of my house (for reasons I'll get too later) & there arent a whole lot of power outlets back there. Granted I could run a power strip and move on with my life but before I lived here I lived in a house with 7 power outlets, total, so whilst I do own a whole lot of power strips, I can't say I'm super enthusiastic about having one on the counter in my laundry room. Also, I had a cough better cough solution in mind. The Pi 3B+ that @confusis hooked me up with from Taiwan had been undervolting during boot since forever & it was hanging off of an old Floureon 5 port PA40W USB charger. One more trip to the IoT store & I had a UCtronics PoE hat & that charger just became surplus to requirements. As a quick aside for anyone thinking of picking up one of the UCTronics PoE hats they do supply 5v3a using 802.3at as opposed to the 5v2.5a using 802.3af of the official PoE board from the Raspberry Pi foundation but the layout of this one means that unlike the official PoE hat if you want to mount a 40mm fan on there, you need an offset adapter. Whilst I haven't tested it a quick eyeoff of Thingiverse has me wondering if this adapter might do the job, or if its time to finally learn some basic OpenSCAD, break out the calipers and just come up with a purpose built adapter. But those are problems for another day. For now the whole thing is back in its case, the UCTronics PoE HAT is compatible with both the rpi-poe device tree overlay and/or the poe_fan_temp dtparam options used by the stock HAT & the whole thing is running rock solid serving as my SDR rig again. Also, I have a USB Power adapter now for the lab-in-a-box.
By now the whole setup was starting to look kind of messy, not that it super matters but keeping it as compact as reasonably possible is a goal for this project. We are also getting fairly close to, well, now, as of when this is being written. A quick trip to ebay saw some m2 threaded rod and a whole bunch of m2 female-female standoffs in a variety of sizes ordered. Along with a couple of other bits & pieces like heatsinks. The threaded rod made it out & i was able to source 100 m2x3mm bolts from a seller in country for like $5 & then COVID kicked in, and much like VTECH, ruined everyones fun yo. One other thing that I did manage to order before the world shut up shop for the forseable future was perhaps the product that satisfied the idea that started this whole thing. But for now, threaded rod, 100 m2x3mm bolts & the whole thing was a little more contained & the situation went from this:
to this:
to this:
Then, as if out of nowhere, some of the "ceramic" heatsinks i ordered arrived so it looks looks like this now:
Already I regret not using a Zero W for the rig with the IoT pHAT on it, mostly because I would like to put some time into learning about nexmon, adding a 5GHz wifi USB dongle & a Logitech Unifying Reciever reflashed witht the rfstorm firmware & putting it to use running Bettercap. Another case of more better when it comes to just having wireless interfaces to use. I may yet get my wish though, just before the whole world closed down, I also managed to finally source a GPS HAT with the PPS output routed to the GPIO header. Back in the day there were a bunch (see here, here & here, ther were a few more too, but meh). Whilst It isnt super hard to use something like the Adafruit Ultimate GPS Breakout & either Dupont cables or even a proto hat or some time in Ki CAD and have a perfectly servicable GPS with PPS board, thats not the DIY project I want to do right now. So , after far too much time on Tindy I found & ordered this.
The plan is to put the Pi Zero W I ordered from here (Everywhere else seemed to have sold out the second COVID started to look like a lockdown) in to replace the non-W currently deployed, finish building that into a Bettetcap machine & then use the non-W board, this GPS HAT and another USB/ethernet pogo pin hat (also ordered but in COVID purgatory) to build a stratum-1 GPS disciplined NTP/PTP clock for my local network. As I slowly upgrade as much of the network as possible to bonded 10Gb links (with RSTP 1G failover links), I've become aware that (PDF) some of the Solarflare NIC's have fairly robust support for PTP & that aside, running my own GPS disciplined NTP server seems like a fun project too. This, ultimatly, is why everything currently lives in the laundry room at the back of the house, its close to windows that have a half decent view of the sky, just in case.
Somewhat longer term I have a couple of proto boards on order from Core Electronics that I'm hoping to pair up with some cheap Chinese enc28j60 modules, its possible to expose a 2nd SPI bus on the GPIO header of a Pi using a device tree overlay, so to my way of thinking, that means its....probably...possible to have two 10Mb/s ethnernet links on top of a Pi zero W....and maybe bond them, put them in a bridge with the USB backed ethernet port and do some RSTP config. I've also read that it might be possible to use two specific USB/ethernet chips to get dual USB ethernet. I'd much rather figure out the what the systemd-networkd & mstpd config will need to look like to do that on something like this than on a box thats being used day in & day out by people who think computering is magic and just works. Once I can get some u.fl - (RP)SMA pigtails made up (i'd super like to use semi-rigid coax because its super nice to handle, but sometimes cost prohibitive) there is a local shop that will laser cut some 3mm acrylic mounts for them that i can bolt on to the stack to enable a wider range of antennas to be used for the IoT pHAT & GPS board There is also the Banana Pi Bpi-M2-Zero, it has a little more CPU power, makes a little more heat, has an external antenna connector, lacks the test points the pogo ping board uses to tap the USB but does have a header on board for 100MB/s ethernet & a "pi compatible" GPIO header. I've also seen docs and reports that appear to suggest its possible to bring up Arch Linux Arm on it using the device tree overlay for the Ubuntu image.
All said & done though, I have one rig to finish configuring, another to start building & Ive reached the limits of what I can do with the current power and networking solutions in place. Whilst I have some thoughts about different ways to solve both, I havent yet settled on any one solution for either. This is likely to be an ongoing project for a fair while though, & not just because the world has ground to a halt because COVID, there are a number of different technologies i'd like to lab, learn & deploy & thus far doing it this way has proven to be alot of fun.
Well technically not quite the beginning, before this there were other iterations going back to an Antec 300 & a bunch of miscellaneous consumer boxes. It is the oldest photo I could find though.
Well, recent anyway
Spanning tree protocol essentially detects loops in a network & disables the looped ports, preventing broadcast storms. Additionally each bridge (think switch) & each path (think a connection between 2 devices) is assigned a cost (usually based on link speed). This means that if there are multiple bridges (switches) then a root bridge can be elected & if there are multiple paths (connections) between bridges then the lowest cost path (the fastest) can be held active whilst the higher cost paths can be disabled (and not become loops in the network at the same time). The root bridge sends out discovery packets periodically & if a path goes away then the next lowest cost path is brought up, if the other bridges don’t see discovery packets from the root bridge over some block of time then the next lowest cost bridge assumes that role. The R in RSTP just basically enhances the protocol to make reconvergance when the network topology changes more Rapid. This all happens at layer 2 & with some planning means the network can failover to links of different speeds (yes, I know LACP, their not mutually exclusive) & handle switch failures.
if you have more than 1 identical link between two devices you can bond them together to behave as one logical link. LACP is a variation of this technology that enables some monitoring & intelligence around load balancing traffic across links and detecting & handling when a link goes down & comes back up.
