Lab-in-a-Box

jØrd

SCSI for life
Original poster
Digital Seppuku
Moderator
Gold Supporter
Bronze Supporter
LOSIAS
Jul 19, 2015
814
1,316
Rich (BB code):
   ▒▒   ▒▒▒▒ ▒▒▒▒▒     ▒▒▒▒▒ ▒▒     ▒▒▒▒     ▒▒▒▒▒  ▒▒▒▒ ▒▒ ▒▒
   ▒▒  ▒▒▛▀▒▒▒▒▛▀▒▒    ▒▒▒▒▒▒▒▒    ▒▒▛▀▒▒    ▒▒▛▀▒▒▒▒▛▀▒▒▖▝▒▒▒▒▛▘
   ▒▒  ▒▒▒▒▒▒▒▒▒▒▒▛▘▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▛▘▒▒ ▒▒ ▒▒▛▘
   ▒▒  ▒▒▛▀▒▒▒▒▛▀▒▒▖▝▀▀▘▒▒▒▒▌▝▒▒▒▌▝▀▀▘▒▒▛▀▒▒▌▝▀▀▘▒▒▛▀▒▒▒▒ ▒▒ ▒▒▒▒
   ▒▒▒▒▒▒ ▒▒▒▒▒▒▒▛▘    ▒▒▒▒ ▒▒    ▒▒ ▒▒    ▒▒▒▒▒▛▘▝▒▒▒▒▛▘▒▒▛▀▒▒▖
   ▝▀▀▀▘▝▀▘ ▝▀▘▝▀▀▀▀▘     ▝▀▘▝▀▘  ▝▀▘    ▝▀▘ ▝▀▘    ▝▀▀▀▀▘  ▝▀▀▀▘ ▝▀▘ ▝▀▘
So now I have had some additional “free” time enforced upon me I figured it was time to write up my ongoing side project.

It all started when I happened across this web page. A Pi Zero with Ethernet & two USB-A ports, what’s not to love!? For the longest time though I had no need for one & shipping from the UK would probably be meh so I just bookmarked it & carried on with life. Somewhere later on down the track I got stuck into overhauling my network, it started with better wifi, then some fibre, better switches, 10Gb, bonding, etc. Along the way I had to learn more about some core networking technologies. Things like LACP, spanning tree and & its myraid variations, IGMP, LLDP, protocol snooping, etc. It’s been a journey that’s still far from being “done”¹ ².

Eventually breaking my home network over & over again started to get super old & how do I even know if the config & hardware I’m bolting together are even working as intended. I needed a test rig, it needed to be portable, easy to power, have a wired network interface (ideally more than one), some wireless interfaces would be nice to have as well & it needed to be easy (for me) to reconfigure or repurpose or nuke & pave or whatever I needed. This began when I wanted to know if the DHCP snooping I had spent far too long learning & deploying was, well, snooping on DHCP. So, perfect excuse, I pulled up that bookmarked RISC OS bits page and put in an order. A couple weeks or so later this landed on my doorstep.

I took a punt that whatever hardware they were bolting on to the Pi Zero to expand the I/O was going to be something normal enough to work with Linux. As I found out later on, this is very much the case so after taking 10 minutes to look nostalgically at RISC OS I dd’d a Linux image onto the MicroSD card, set to work configuring dhcpd & plugged it into just about every piece of networking hardware I could find to see if it was able to hand out leases or not. Using a combination of device logs, dhcpdump & nmap I got the answers I was looking for. Success! and validation of the idea that small, low power hardware would be a viable approach to labbing. This philosophy is somewhat stolen from MikroTik, with a few exceptions a $30 hAP mini will do most of the same things, using the same OS, as a $300 CRS 3 series switch or a $1,000 CCR router, usually just slower or with lower maximums or the like. Validating that I could apply the same style of approach to my own needs was super satisfying, my rouge DHCP server didn’t need to be able to handle 1,000 clients, it just needed to poke whatever it was connected too until it got a response.

So cool, now I have a Pi Zero with Ethernet & two USB-A ports, it’s useful, but it lacks any wireless capabilities & has a 40 pin GPIO header going unused. Mostly in parallel to rolling out DHCP snooping I had also been configuring RSTP across my whole network properly³. This set me thinking, as part of upgrading what I can to dual 10Gb connections using LACP⁴ I now have a bunch of hardware that has 1Gb onboard Ethernet ports going unused, if I can get my head around configuring RSTP on Linux then I could use those ports to provide redundant links between those devices and the 1Gb switch in my rack, should both of the links in the 10G LAGG go down or the 10G switch itself. A quick google later lead me to the Pi Jack. It’s basically an enc28j60 Ethernet - SPI bridge hooked up the the GPIO header on the Pi & a device tree overlay to configure the pins. It’s also been out of production since forever. No matter though, The enc28j60 is used everywhere and the clone boards that came after are arguably better, allowing for user config of slave selection using jumpers. A quick trip to Amazon (or Ebay or Aliexpress) and a spare hAP lite later & I had a dual ethernet Raspberry Pi Zero. Granted the enc28j60 board ran at all of 10Mb/s & the dual USB/ethernet board that connected using pogo pins to the test pads on the underside of the Pi would do 100Mb/s & used the same chipset as the Pi 2 to get there but no matter, I didn't need speed, I needed interfaces. For funsies there is a hAP mini on top of the fridge in the kitchen thats acting as the near side of an nstream bridge & the hAP lite as the far side. This is potentially the least efficient way to do this vs just using wifi, but meh, its more "fun".


Ultimately this ended up being a fantastic platform to learn the specifics of (R)STP on linux with my eventual solution being to use the MSTPd userspace daemon and NetworkManager-dispatcher to trigger a script to handle runtime configuration of the bridge as the interface came up. Somewhere around this time I also moved to a new ISP that provided native IPv6 and after learning a whole bunch more mind melting weirdness about that protocol, used this rig as my test bed to migrate from NetworkManager (using DHCPCd, dnsmasq, chrony, wpa_supplicant & NetworkManager-dispatcher) to systemd-networkd, systemd-timesyncd, systemd-resolved, iwd & networkd-dispatcher. At this point I figure I have gotten my moneys worth out of this board, using it as a test bed means migrating the rest of my network, the bits that actually matter, has gone substantially smoother.... But, it doesnt have WiFi, or Bluetooth, it doesn't need either buuuuuut I mean, it would be kind of cool. A quick trip to Core Electronics solved problem in short order.

For a while everything was good, but then I got to thinking, I have 2 more ethernet ports on that hAP and a spare Pi zero. A bit of google sleuthing around lead me to this Tindie page, Kind of looks alot like this is where RISC OS Bits are getting their boards from, maybe. Regardless, one order later & another one was in the post, thats the wired bit solved. Next up was the wireless, There have been a number of wifi boards back in the day for the Pi Zero, but I jumped on this hype train far too late to source any of the early ones (see here, here or here for a few examples). There was one name that kept coming up though, the now defunct Red Bear electronics and their IoT pHAT. The company may be gone but as long as there are still boards for sale & a git repo to clone it looks like a viable option. Hitting up The IoT Store saw one on my doorstep pretty quickly and a trip to ebay landed me some nicely color coded GPIO headers.


A little patience, some filthy soldering and a bit of luck later & my second lab rig was beginning to look like ready

Not much later The USB/ethernet board arrived and this rig was built


But with every problem solved, a new one is often created. This whole lab lives in the corner of the laundry room in the back of my house (for reasons I'll get too later) & there arent a whole lot of power outlets back there. Granted I could run a power strip and move on with my life but before I lived here I lived in a house with 7 power outlets, total, so whilst I do own a whole lot of power strips, I can't say I'm super enthusiastic about having one on the counter in my laundry room. Also, I had a cough better cough solution in mind. The Pi 3B+ that @confusis hooked me up with from Taiwan had been undervolting during boot since forever & it was hanging off of an old Floureon 5 port PA40W USB charger. One more trip to the IoT store & I had a UCtronics PoE hat & that charger just became surplus to requirements. As a quick aside for anyone thinking of picking up one of the UCTronics PoE hats they do supply 5v3a using 802.3at as opposed to the 5v2.5a using 802.3af of the official PoE board from the Raspberry Pi foundation but the layout of this one means that unlike the official PoE hat if you want to mount a 40mm fan on there, you need an offset adapter. Whilst I haven't tested it a quick eyeoff of Thingiverse has me wondering if this adapter might do the job, or if its time to finally learn some basic OpenSCAD, break out the calipers and just come up with a purpose built adapter. But those are problems for another day. For now the whole thing is back in its case, the UCTronics PoE HAT is compatible with both the rpi-poe device tree overlay and/or the poe_fan_temp dtparam options used by the stock HAT & the whole thing is running rock solid serving as my SDR rig again. Also, I have a USB Power adapter now for the lab-in-a-box.


By now the whole setup was starting to look kind of messy, not that it super matters but keeping it as compact as reasonably possible is a goal for this project. We are also getting fairly close to, well, now, as of when this is being written. A quick trip to ebay saw some m2 threaded rod and a whole bunch of m2 female-female standoffs in a variety of sizes ordered. Along with a couple of other bits & pieces like heatsinks. The threaded rod made it out & i was able to source 100 m2x3mm bolts from a seller in country for like $5 & then COVID kicked in, and much like VTECH, ruined everyones fun yo. One other thing that I did manage to order before the world shut up shop for the forseable future was perhaps the product that satisfied the idea that started this whole thing. But for now, threaded rod, 100 m2x3mm bolts & the whole thing was a little more contained & the situation went from this:

to this:

to this:

Then, as if out of nowhere, some of the "ceramic" heatsinks i ordered arrived so it looks looks like this now:



Already I regret not using a Zero W for the rig with the IoT pHAT on it, mostly because I would like to put some time into learning about nexmon, adding a 5GHz wifi USB dongle & a Logitech Unifying Reciever reflashed witht the rfstorm firmware & putting it to use running Bettercap. Another case of more better when it comes to just having wireless interfaces to use. I may yet get my wish though, just before the whole world closed down, I also managed to finally source a GPS HAT with the PPS output routed to the GPIO header. Back in the day there were a bunch (see here, here & here, ther were a few more too, but meh). Whilst It isnt super hard to use something like the Adafruit Ultimate GPS Breakout & either Dupont cables or even a proto hat or some time in Ki CAD and have a perfectly servicable GPS with PPS board, thats not the DIY project I want to do right now. So , after far too much time on Tindy I found & ordered this.

The plan is to put the Pi Zero W I ordered from here (Everywhere else seemed to have sold out the second COVID started to look like a lockdown) in to replace the non-W currently deployed, finish building that into a Bettetcap machine & then use the non-W board, this GPS HAT and another USB/ethernet pogo pin hat (also ordered but in COVID purgatory) to build a stratum-1 GPS disciplined NTP/PTP clock for my local network. As I slowly upgrade as much of the network as possible to bonded 10Gb links (with RSTP 1G failover links), I've become aware that (PDF) some of the Solarflare NIC's have fairly robust support for PTP & that aside, running my own GPS disciplined NTP server seems like a fun project too. This, ultimatly, is why everything currently lives in the laundry room at the back of the house, its close to windows that have a half decent view of the sky, just in case.

Somewhat longer term I have a couple of proto boards on order from Core Electronics that I'm hoping to pair up with some cheap Chinese enc28j60 modules, its possible to expose a 2nd SPI bus on the GPIO header of a Pi using a device tree overlay, so to my way of thinking, that means its....probably...possible to have two 10Mb/s ethnernet links on top of a Pi zero W....and maybe bond them, put them in a bridge with the USB backed ethernet port and do some RSTP config. I've also read that it might be possible to use two specific USB/ethernet chips to get dual USB ethernet. I'd much rather figure out the what the systemd-networkd & mstpd config will need to look like to do that on something like this than on a box thats being used day in & day out by people who think computering is magic and just works. Once I can get some u.fl - (RP)SMA pigtails made up (i'd super like to use semi-rigid coax because its super nice to handle, but sometimes cost prohibitive) there is a local shop that will laser cut some 3mm acrylic mounts for them that i can bolt on to the stack to enable a wider range of antennas to be used for the IoT pHAT & GPS board There is also the Banana Pi Bpi-M2-Zero, it has a little more CPU power, makes a little more heat, has an external antenna connector, lacks the test points the pogo ping board uses to tap the USB but does have a header on board for 100MB/s ethernet & a "pi compatible" GPIO header. I've also seen docs and reports that appear to suggest its possible to bring up Arch Linux Arm on it using the device tree overlay for the Ubuntu image.

All said & done though, I have one rig to finish configuring, another to start building & Ive reached the limits of what I can do with the current power and networking solutions in place. Whilst I have some thoughts about different ways to solve both, I havent yet settled on any one solution for either. This is likely to be an ongoing project for a fair while though, & not just because the world has ground to a halt because COVID, there are a number of different technologies i'd like to lab, learn & deploy & thus far doing it this way has proven to be alot of fun.

Well technically not quite the beginning, before this there were other iterations going back to an Antec 300 & a bunch of miscellaneous consumer boxes. It is the oldest photo I could find though.
Well, recent anyway

Spanning tree protocol essentially detects loops in a network & disables the looped ports, preventing broadcast storms. Additionally each bridge (think switch) & each path (think a connection between 2 devices) is assigned a cost (usually based on link speed). This means that if there are multiple bridges (switches) then a root bridge can be elected & if there are multiple paths (connections) between bridges then the lowest cost path (the fastest) can be held active whilst the higher cost paths can be disabled (and not become loops in the network at the same time). The root bridge sends out discovery packets periodically & if a path goes away then the next lowest cost path is brought up, if the other bridges don’t see discovery packets from the root bridge over some block of time then the next lowest cost bridge assumes that role. The R in RSTP just basically enhances the protocol to make reconvergance when the network topology changes more Rapid. This all happens at layer 2 & with some planning means the network can failover to links of different speeds (yes, I know LACP, their not mutually exclusive) & handle switch failures.
if you have more than 1 identical link between two devices you can bond them together to behave as one logical link. LACP is a variation of this technology that enables some monitoring & intelligence around load balancing traffic across links and detecting & handling when a link goes down & comes back up.
 

jØrd

SCSI for life
Original poster
Digital Seppuku
Moderator
Gold Supporter
Bronze Supporter
LOSIAS
Jul 19, 2015
814
1,316
Rich (BB code):
   ▒▒   ▒▒▒▒ ▒▒▒▒▒     ▒▒▒▒▒ ▒▒     ▒▒▒▒     ▒▒▒▒▒  ▒▒▒▒ ▒▒ ▒▒
   ▒▒  ▒▒▛▀▒▒▒▒▛▀▒▒    ▒▒▒▒▒▒▒▒    ▒▒▛▀▒▒    ▒▒▛▀▒▒▒▒▛▀▒▒▖▝▒▒▒▒▛▘
   ▒▒  ▒▒▒▒▒▒▒▒▒▒▒▛▘▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▛▘▒▒ ▒▒ ▒▒▛▘
   ▒▒  ▒▒▛▀▒▒▒▒▛▀▒▒▖▝▀▀▘▒▒▒▒▌▝▒▒▒▌▝▀▀▘▒▒▛▀▒▒▌▝▀▀▘▒▒▛▀▒▒▒▒ ▒▒ ▒▒▒▒
   ▒▒▒▒▒▒ ▒▒▒▒▒▒▒▛▘    ▒▒▒▒ ▒▒    ▒▒ ▒▒    ▒▒▒▒▒▛▘▝▒▒▒▒▛▘▒▒▛▀▒▒▖
   ▝▀▀▀▘▝▀▘ ▝▀▘▝▀▀▀▀▘     ▝▀▘▝▀▘  ▝▀▘    ▝▀▘ ▝▀▘    ▝▀▀▀▀▘  ▝▀▀▀▘ ▝▀▘ ▝▀▘
There has been progress and there has been setbacks. During a moment of what can only be described as complete stupidity on my part, whilst reassembling the stack and pushing a hat back onto one of the pi's i managed to physically crack an SD card. I would have cared far less if it were the SD card in the top Pi, which is in a much more bare-bones state as it stands, but it wasn't, it was the card in the bottom Pi, which i had sunk a fair bit of time and effort into configuring. This outright murdered momentum for the project for a bit, besides from being a reference implementation for a bunch of tech im implementing across the rest of my network i also had other potential plans for the device that are now well and truly on the back burner for a bit.

That being said, all hope is not lost, i reflashed a new card with a base image tonight with the intention of, at some point, rebuilding it. And before anyone makes noise about backups, yes, i know, i run Bacula on basically everything, it was on the to-do list, i just didn't get to it in time 😣. So be it though, onward to more interesting problems to solve. Some parts I ordered from Core Electronics came in recently & create as good an oppertunity as any to practise my soldering skills (yes, i have none, much like installing Bacula on the lab gear, its on the to-do list). Also, a quick shout out to the team there, the breadboard that I had ordered was out of stock & after explaining my needs to them they just reuped me to a board they had in stock that met the same needs and ate the cost difference. Ultimately the goal is likely to take the Adafruit board and solder a couple pairs of sockets to it to take two Olimex ENC28j60 modules. They claim to be the smallest on the market, they are also kind of expensive though. To that end, i have ModMyPi protoboard that i will solder sockets too and try out a pair of substantially cheaper W5500 modules on. As best as I can tell the drivers exist in kernel for them so ¯\_(ツ)_/¯




And whilst I am still waiting on one final Pi Zero W to make it from over east I did take the opportunity to solder all the headers to my GPS hat. The PPS one is the only one I will actually use but I put the rest of the headers on because why not.


That kind of covers progress up to now, I have a bunch of M2 fittings that may or may not ever leave postal purgatory along with some ceramic heatsinks, couple of bits & pieces stuck in the back of an AusPost van for the foreseeable future & an OS image to rebuild. Two steps forwards and one step back perhaps.
In an unrelated win though I got my old Sony CCTV camera running, it runs off PoE, its SD resolution & hot-as-hell but the sensors in Sony pro CCTV gear are fantastic so I have a couple saved ebay searches for when model numbers come up for cheap. So for an hour or so today this was my Twitch streaming rig. I have a couple of other, older, Sony CCTV cameras that dont output a network stream (composite over BNC vs the MJPEG on the SNC-CS11 pictured). Having done some web cam modding in the past (and ironically not being able to find a picture of any of it) and also having two (soon 3) Pi zero's with camera connectors, a Sony SSC-E443P & somewhere a Sony (iirc) SSC-E473P the temptation to grab a couple of cheap easycap adapters and get bk into camera/video is super there, maybe just finish some other projects 1st or something 😩
 
  • Like
Reactions: Soul_Est

jØrd

SCSI for life
Original poster
Digital Seppuku
Moderator
Gold Supporter
Bronze Supporter
LOSIAS
Jul 19, 2015
814
1,316
,-.           _                _                                 ,----.
| |          / |              (_)                                | ,-. \
| |          | |                                                 | | | |
| |          | |                                                 | | | |
| |    ,----.| |,-.           ,-. ,---.          ,----.          | | | | ,---. .-.  ,-,
| |    \,--. \| ,-. \         | || ,-. \         \,--. \         | '-'_/| ,-. | \ \/ / 
| |     ,-.| || | | | ,----'\ | || | | | ,----'\  ,-.| | ,----'\ | ,-. \| | | |  \  /  
| |    / ,-. || | | | \,----' | || | | | \,----' / ,-. | \,----' | | | || | | |  /  \  
| |____| \_| || |_; |         | || | | |         | \_| |         | |_/ || |_| | / /\ \ 
\_____/'.__,_/\_,__,'         \_|\_| |_/         '.__,_/         |____,''.___,'/_/  \_|

So some progress has finally been made with this, its still slow going but that's somewhat unavoidable for the time being. To kick off, the M2 thumb nuts I ordered a while back arrived on the doorstep. This means that the whole assembly is somewhat level again instead of rocking ever so slightly corner-to-corner, the lower USB ports are lifted up just a little bit so larger connectors/thumb drives/etc will mate up better now & the razor sharp ends of the threaded rod I snapped off aren't carving up everything they sit on top of. Also, it just looks better.

Around about the same time the UUGear eMMC μSD cards arrived. These dont make all that much sense from a strictly cost based perspective but my hope is that they will hold up a little better as OS drives than traditional μSD cards. Either way though, after cracking one of the cards accidentally I had to do something, and besides from looking cool these ones are much harder to not notice when (dis)assembling the hardware.



That is, for better of worse, about the extent of the hardware progress, its slow going. The software situation has improved substantially though. I didn't have a backup of the μSD card that got cracked & it took a while to get motivated to sink that effort in a second time. With replacement cards to hand though it seemed like as good a time as any to sit down and make a start. There are still a lot of loose ends to tie up, enabling the hardware RNG watchdog come top of mind but mDNS/DNSsd, rkhunter, smokeping, /etc/machine-info, bacula and on and on are all still to be handled before its really in a sensible state to start adding services on top. Networking is done though, its nothing super fancy but with two ethernet interfaces (both running at different speeds) this machine was (and is again) my reference RSTP platform. Whilst many a seasoned network admin will have a laundry list of entirely correct reasons to disparage any of the spanning tree protocols, being able to survive a full switch failure, or the loss of both links in a LACP bond and still stay up, even if its over a less performant link, is super satisfying, especially when the only real cost to implement it is time and effort. Normally STP is well supported in Linux, but RSTP seems to be less so. In the end I settled on using mstpd and doing runtime configuration with networkd-dispatcher. Needless to say, rebuilding this box was a salient reminder as to why i had a reference implementation to begin with.

Ultimately with either the 100Mb/s or both the 100Mb/s and 10Mb/s link(s) connected, the fastest is held up and the other is disabled (so no broadcast storms) and after accounting for the unspectacular wireless bridge between the switch these devices are on and the rest of my network and the processing overhead, etc of the setup on fairly anemic hardware i can smash out almost 70Mb/s in iperf. With the 100Mb/s link disconnected the 10Mb/s is unblocked and picks up as soon as the network has reconvereged on the new topology, performance here is markedly worse though, averaging 4Mb/s. Apparently the enc28j60 is a somewhat minimal hardware implementation, depending alot of software to make up the shortfall, and overhead, and the wireless bridge. Performance wasn't the priority here though, if it were then a Pi Zero would not be the platform of choice but whatever, being able to do this and not having to make any accommodations whatsoever makes me substantially happier than it reasonably should.


Unfortunately I haven't had anywhere near as much success with the other Pi in the stack. Connecting up the wifi/bluetooth hat to a Pi Zero W sees the onboard wifi & bluetooth just disappear completely. Posting on the Raspberry Pi forum proved fruitless, by the time my post cleared the new user moderation queue it was already 2 pages deep, no one ever saw it. I'm sure the answer is fairly simple, I'm just missing something that is obvious to anyone who has a better understanding of how the various buses on the Pi are provisioned and broken out. I dunno, that one is on the back burner whilst the other Pi still needs some time invested into it. If anyone here knows the answer, the hat uses an SDIO and UART bus. Success in steps I guess. No progress has been made adding the third Pi in either, Australia Post shitcanned the USB/eth board i ordered for it, that set this one back a fair bit.