I’ve been doing a lot of work on various ARM things. A lot of it has been in support of making community builds of Chef for other folks, but I have a new goal: providing ARM nodes to the Internet at large as a hosting option.
I’m not going to go into a lot of details here yet, as I’m saving a lot of that content for my hosting company’s blog. I’ll announce it here when it goes live. There are a few things in the way, but A lot of the deep dive technical stuff will be posted here. So, now that I’ve gotten that out of the way let’s get into the usual ARM update:
RK3368 – It’s been fun
First the bad news: I’m giving up on the RK3368 as an ARMv8 platform for now. Part of this is the kernel mess (but things are looking better for 4.6), but most of it is availability of hardware (low), the (relative) high cost of the hardware, the focus of Android on this SOC, and the common implementation of the serial port connected to the MMC/SD/TF slot. Finding RK3368 hardware is somewhat difficult, and when I do it’s always an Android Media Player/STB. Getting these to run anything else is..really hard. Rockchip doesn’t support upstream das u-boot, and while the #linux-rockchip community on freenode is making strides, they are focused on the much more available ARMv7 variants instead of the ARMv8 RK3368. It’s understandable, and I’m not at all upset. The reality of the situation is that the RK3368 based systems I can find are much more expensive than other ARMv8 solutions. That then just leaves the Serial Port on MMC/SD/TF slot issue, which is a real deal breaker for me. In a 100% remote environment, OOB console is required and for most things that means IPMI or Serial. IPMI is out on most ARM devices so that means serial. USB isn’t great for primary storage (for a lot of reasons), and having the storage card share pins with the serial console just means it’s a disaster waiting to happen. I had high hopes for the Geekbox, but even that turned out to be a real disappointment. The focus for this SOC is obviously Android, and that’s fine, but it doesn’t suit my needs at all.
On the GeekBox
I don’t want to rag on them too hard, it’s a lovely piece of kit. But it suffers from some shortcomings. Stock and availability of components being one of them. I’ve placed several orders for them, and they never ship on the advertised time frame. The folks at Geek Buying are very nice, and very transparent about supply issues, but it’s still been rather frustrating. Then there is the odd selection of accessories. The docking station is mostly a USB hub and easier exposure of the edge connector functions, but there is no case for it. The CPU fan accessory does nothing and doesn’t have a reasonable mount even if you want to use it (you do, I’ve melted the case by accident on one of mine doing compiles). The lack of case when the board is in the docking station caused me to loose another board due to static discharge. Then there is the software: android runs fine. No major issues there. The Linux side however, is a bit of a mess. It’s a customized version of Ubuntu 14.04’s unsupported arm64 port. Most vendors seem to be taking this route, and I don’t understand why. The real problem: the video is not stable. The signal is solid (it’s digital after all), but the picture has waves going up and down the screen on every monitor I’ve tried. Android is fine, it has something to do with the Linux kernel they provide. Makes it very hard to even get the DHCP IP address and then drop out to a console. In fact, a FB console is not even supported. X or nothing. I chose nothing.
Again, this isn’t a bad device, just doesn’t suit my needs.
This device has been a breath of fresh air. Also ARMv8 (based on an amlogic 905S SoC) but only 4 cores (compared to the RK3368’s 8 cores). No onboard storage, you have to add a MicroSD/TF card. The best part: broad community support. This SoC just landed in 4.6 and the makers of the board (HardKernel out of Republic of Korea) are working with Amlogic to get full support upstream in mainline. Right now, mine are still running their heavily patched 3.14 kernel – but the source and configs are readily available and adding support for things like LXC was very simple. I’m running an “odrobian” install, which is a minimal Debain Jessie root on an ext4 filesystem and some kernel config tweaks for Docker support. I’ve rolled my own kernel to add the rest of the configuration options needed for LXC and I’m running that for my build and test nodes.
I also just deployed four of these into production: two for recursive name servers, and two for authoritative name servers. They are cheap ($40 from HardKernel, and there are worldwide resellers) and they are case compatible with RPi B rev 2. They also appear to sip power. I picked up a USB Meter from Adafruit, and I can’t get them above 0.3A at 5V for most of my testing. Now, I don’t entirely believe those numbers (this device has a very low sample rate), so I’m going to do some more testing with a NI VirtualBench at TechShop this weekend to see if I can learn more. I’ll post more details on that once I have them. For now, I’m being ultra conservative and deploying with a PSU that can support six devices up to [email protected] each.
So far, these boards are great. Biggest thing to be aware of: when you buy one, get their power cord. It’s a 2.5mm barrel connector (HardKernel makes and sells pigtails and USB adapters). Those connectors are apparently common everywhere but here.
The crappy cell phone chargers we use for MicroUSB power everywhere are usually not enough for most of these devices (I’ve killed more than a couple RPi boards this way), so get a good solid power supply for more than $3. The ones sold by SparkFun and Adafruit are well tested and good if you need basic MicroUSB power, they also offer good all around 5V supplies for general use. If you have several devices, IKEA makes an inexpensive 3 port USB charger that I’ve been using without issue, or the Anear power adapter I’m using in a few places with up to six devices pulling [email protected] each seems to be working very well also.
I’ve recently pulled mine out of mothballs for a few reasons:
- They run a mainline kernel now, so they are great for a serial console server which cheap USB-Serial adapters
- I need an inexpensive but reasonably accurate NTP server fed from GPS
These are descent little boards, with a Dual Core ARMv7 Allwinner A20 proc and 1GB of RAM. They also have onboard eMMC, but the TF/MicroSD is better supported. They also have native 1Gb Ethernet (instead of a USB interface like the RPi products).
The Armbian project supports these (along with most commonly available Allwinner based boards), with a descent Jessie rootfs and either a recompiled vendor kernel or a vanilla upstream kernel. I’m using the vanilla for my console server, and the vendor kernel for the GPS stuff for now. I want to move to the vanilla kernel for that as well, but there are Device Tree issues to work out.
These are also neat because they have Arduino compatible headers, but at a different voltage (3.3v vs the normal Ardunio of 5v). The good news is that there is a level conversion shield, the bad news is that it doesn’t quite fit on the nano (the Ethernet jack is in the way), so I’ve needed to get some header extensions to plug it in completely. Once I get the headers extensions, I plan on stacking an Adafruit GPS Logger Shield w/ Module and then a Adafruit I2C Controlled Keypad and display shield on top. I’ve already tested that the GPS module is compatible with gpsd, and with a vanilla kernel I should be able to get the 1PPS off the shield and into gpsd and ntpd for an inexpensive Stratum 1 time source. I’ll update here will the full build out and configuration once I’ve got it all/mostly working.
I don’t have any yet, but I’m excited. Quad Core ARMv8, 2GB RAM, under $30. I ordered 22 of these through their Kickstarter. I hope to have them by the end of the month, and they will be the basis for Gen 1 of my hosting product. I’ll be talking a lot more about these once they arrive and I start testing and design for support hardware.
Huskyboard / LeMaker Cello
I’m really hoping that 2016 is the year of AMD’s comeback for many reasons. Either way, the A series Opteron has always caught my eye. The Huskyboard is still “Soon” but the LeMaker Cello is shipping in the next couple months, looks to be nearly identical specs wise, and comes in at $299. I’ve ordered two for testing and might use one to be my new mail server since I can add 2 SSDs over SATA to them. Expect more on these in the future as well. I just wish either of them shipped with headers for the second onboard gigabit NIC.
On SD Cards
Typically, I’ve been buying the cheapest 16G SD cards I can get my hands on from Fry’s. These are usually name brand (PNY, Toshiba, Patriot, etc) and they have been “reasonable”. With the goal of making a commercial hosting product with these, “reasonable” probably won’t fly with paying customers. So, I did some research and everything was pointing in one direction: Samsung EVO. So, I went back to Fry’s and run some tests with iozone3. I’m including the results below:
- Toshiba 16GB on PcDuino 3 Nano Armbian 5.0
- Samsung EVO 64GB on PcDuino 3 Nano Armbian 5.0
- missing files, will re-run and update once I get another EVO 64GB
- Samsung EVO+ 32GB on PcDuino 3 Nano Armbian 5.0
- Toshiba 16GB on ODROIC-C2 odrobian Jessie (kernel3.14.29-14-odrobian+)
- Samsung EVO 64GB on ODROIC-C2 odrobian Jessie (kernel3.14.29-14-odrobian+)
- Samsung EVO+ 32GB on ODROIC-C2 odrobian Jessie (kernel3.14.29-14-odrobian+)
All of the tests were run on the device while connected over a serial console, using the default EXT4 filesystem on the SD card directly. No other activity was being run on these devices at the time, but they were connected to a local LAN with (no WiFi). Here is the exact iozone3 command used:
iozone -az -i 0 -i 1 -e -R -+T -b ~/output.xls | tee ~/output.log
Power wise, it looks like the Samsung EVO is the big winner. It appeared to consume less power than the other cards. Again, my power measurements are inaccurate and I hope to address that this weekend. Beyond that, the Toshiba is the clear looser but there doesn’t seem to be much difference between the EVO and EVO+ on these boards (probably due to limitations of the MMC interface of the SOC). So, I’m sticking with the EVO cards when I care about performance. I also don’t have a 32GB EVO card or a 32GB Toshiba to do a direct 1:1 comparison. I hope to fix that before I get the Pine64 boards. The price/performance curve definitely has the EVO in it’s favor, where the bulk of the 64GB MicroSD cards on Amazon are (as of 24MAR2016) around $15, the EVO is $20 and the EVO+ is $25. These tests show that the EVO+ doesn’t appear to be worth the extra 20% price increase for these devices.
I think that’s a good update for now, I’ll post more as I have it!