Back to post index

Spinning up smyslov the firewall
Tags: [OpenBSD] [router] [firewall]
Published: 09 Nov 2013 17:20

Post summary

The first part of this post explains the steps that I took to set up an old Sun Blade 100 that I had lying around as an OpenBSD router / firewall / PPPoE client. Once set up, I experienced problems with my PPPoE connection frequently disconnecting - the second part of this post explains how I diagnosed the problem to be a kernel level PPPoE bug in OpenBSD and FreeBSD (because they share roughly the same source for kernel level PPPoE).

Table of contents:

Step 0 - time for change

A Linksys WRT54GL running the Tomato firmware has served my home network for a long time but I started seeing my persistent connections disconnect regularly. The disconnects could have been blamed on the modem but looking at the WRT54GL’s uptime showed that it had also reset at the time of the disconnect. I thought that a replacement was in order - the WRT54GL was almost 5 years old at this point.

At around this time, I was also concerned about the security of this setup. I never had any problems with the Tomato firmware, but it bothered me that it was running such an old version of Linux and all the userland tools:

Linux unknown 2.4.20 #1 Sun Nov 29 06:53:09 PST 2009 mips GNU/Linux

As well, around this time, I had read a few posts on Hacker News about commodity routers being trivial to circumvent and gain access to. Because of this, I decided to build my own router.

Before switching to a new router, I ran a small benchmark speedtest. This is not terribly important, but it’s important to note that this test was performed over the wireless interface.

I unplugged everything, and plugged my laptop directy into the modem to check its settings:

02 days 17:21:10 (since last boot)
  xDSL linestate up
  (downstream: 4032 kbit/s, upstream: 800 kbit/s;
   output Power Down: 19.0 dBm, Up: 11.5 dBm;
   line Attenuation Down: 48.0 dB, Up: 29.0 dB;
   snr Margin Down: 8.0 dB, Up: 13.0 dB)

Looks ok, but the stats are not great. 48 dB of attenuation is not good, and neither is 8 dB of SNR margin, according to this page on DSL line stats and their interpretation.

Step 1 - router hardware initialization

I chose to use an old Sun Blade 100 that I picked up at the Waterloo Surplus Sale for $10 a while ago as the router hardware. It doesn’t have great specifications, but a router doesn’t necessarily have to:

Processor: UltraSPARC IIe @ 500 MHz
RAM:       1.5 GB ECC
HDD:       20 GB

One neat feature of the OpenBoot firmware available on these Sun boxes is that it can perform diagnostics. I made sure to perform a diagnostic test on all the hardware before beginning this process, but unfortunately there is no equivalent to memtest86+ on SPARC systems.

I installed latest version of OpenBSD SPARC64 (5.3 at the time of this writing) from The instructions for netbooting the Sun Blade 100 are found at the INSTALL.sparc64 readme.

Note: if using old Sun hardware, make sure your machine is compatable with the OpenBSD/Sparc64 set.

In INSTALL.sparc64, go to the Net Boot or Diskless Setup Information: section. Also helpful:

For my Debian system, the following packages were required:

apt-get install bootparamd rarpd nfs-kernel-server

Note: I already had dnsmasq installed and listening on my ethernet port eth0, handing out IP leases in the range.

Here is how I set up the relevant files on fischer:


sunblade100 root= gateway=

/etc/hosts: sunblade100


tftp     dgram udp wait  nobody  /usr/sbin/tcpd  /usr/sbin/in.tftpd -s /srv/tftp


/srv/tftp/ sunblade100(ro,sync,no_subtree_check)
/srv/nfs/ sunblade100(ro,sync,no_subtree_check)


00:03:ba:18:26:9b sunblade100

In the directory /srv/tftp/, only two files are required:

1) A file named after the hex format of the Sun box’s IP which points to the netboot file:

soft link: 0A000002 -> sparc/OpenBSD/5.3/

2) the OpenBSD kernel:

soft link: bsd.rd -> sparc/OpenBSD/5.3/bsd.rd

Once this configuration is done, bring up the interface:

root@fischer:/srv/tftp# ifup eth0=eth0router
root@fischer:/srv/tftp# echo 1 > /proc/sys/net/ipv4/ip_forward

Boot the Sun Blade 100. At the ‘ok’ prompt, type:

boot net bsd.rd

And follow the installation instructions.

Step 2 - configuring OpenBSD

First step:

~/.profile: export "PKG_PATH="
# pkg_add -v vim
# pkg_add -v screen

One of the first things that I did was md5 all files from root - to note what files I changed, and later for intrusion / rootkit detection:

# find / -type f -print0 | xargs -0 md5 | tee rootmd5_`date +"%s"`;

The functionality that this router has to provide is as follows:

  1. Connect to the internet through my DSL modem using PPPoE.
  2. Provide a wired and wireless connection.
  3. Firewall and NAT between the internet and my LAN.

Connect to the internet through my DSL modem using PPPoE.

The Sun Blade 100 comes with an onboard 10/100 LAN port which I configured to be the PPPoE device (device was labelled gem0 in dmesg):


inet NONE \
  pppoedev gem0 authproto pap \
  authname 'xxx' authkey 'xxx' up
!/sbin/route delete default
!/sbin/route add default -ifp pppoe0
!/sbin/pfctl -e -f /etc/pf.conf



My /etc/hostname.pppoe0 follows the man 4 pppoe configuration almost exactly, with the exception of the following lines:

!/sbin/route delete default
!/sbin/pfctl -e -f /etc/pf.conf

which perform the following additional configurations:

Reboot to reconfigure network and confirm that the connection works with ifconfig pppoe0. It should have information filled in:

# ifconfig pppoe0
        priority: 0
        dev: gem0 state: session
        sid: 0x156c PADI retries: 3 PADR retries: 0 time: 04:55:18
        sppp: phase network authproto pap authname "xxx" 
        groups: pppoe egress
        status: active
        inet --> nn.nn.nn.nn netmask 0xffffffff

Ping out to to test that your routes are set up correctly.

Check hostname resolution with

# ping

Initially, hostname resolution didn’t work for me. If this doesn’t work for you, you will have to manually add public DNS IPs to your resolv.conf file.



Provide a wired and wireless connection.

The relevant interfaces here: athn0, a PCI wireless card, and dc0, a PCI 10/100 ethernet card.


  media autoselect
  mode 11g
  mediaopt hostap
  nwid fieldnotes
  wpakey "xxx"
  chan 11



This sets up two different subnets for my LAN. You don’t necessarily have to do this, I just thought it would be neat. One thing that is possible with separate subnets is to limit or restrict traffic between the wired and wireless interfaces. I didn’t do this.

Next, configure dhcpd, the dhcp daemon that ships with OpenBSD:


option domain-name "my.domain";

shared-network wired {
  # dc0
  subnet netmask {
    option routers;
    option domain-name-servers;

shared-network wireless {
  # athn0
  subnet netmask {
    option routers;
    option domain-name-servers;

Then, reboot. Again, check that things are working with ifconfig athn0 and ifconfig dc0.

Connecting to the wireless or wired interfaces at this point should work - clients should receive an IP through the normal DHCP request methods.

Firewall and NAT between the internet and my LAN.

To do this, edit pf.conf:


ext_if = pppoe0
int_ifs = "{ dc0, lo0, athn0 }"

unroutable = "{,,, \
      , }"

wlan_net = ""
lan_net = ""

set loginterface $ext_if
set limit states 10000
set limit frags 500

# do not filter loopback interface
set skip on lo0

# scrub pppoe mss
match on $ext_if scrub (random-id max-mss 1440 reassemble tcp)

antispoof for lo0

# nat
match out on $ext_if inet from $wlan_net to any nat-to ($ext_if)
match out on $ext_if inet from $lan_net to any nat-to ($ext_if)

# rules
block return log on $ext_if all

# from pf.conf man page, examples section
block in from no-route to any
block in from urpf-failed to any
block in quick on $ext_if from any to
block in log quick on $ext_if from $unroutable to any
block out log quick on $ext_if from !($ext_if) to any

pass quick on $int_ifs all

pass out on $ext_if inet proto icmp from ($ext_if) to any icmp-type 8 code 0
pass in  on $ext_if inet proto icmp from any to ($ext_if) icmp-type 8 code 0

## UDP
pass out on $ext_if inet proto udp from ($ext_if) to any

## TCP
pass out on $ext_if inet proto tcp from ($ext_if) to any

A couple of rule blocks here are directly from man pf.conf page, and others are from some pf.conf example pages I found after searching:


None of this will work unless OpenBSD is instructed to actually forward packets:

sysctl net.inet.ip.forwarding=1

To make this a permanent setting, modify /etc/sysctl.conf by uncommenting the following line:


Once this is done, you should be able to access the internet from clients.

One test that I performed is to nmap my IP from an external machine. This will make sure that all ports are closed and firewalled correctly. Alternatively, visiting a web site that scans your IP like Shields Up! will also tell you that your firewall is configured correctly.

Final configuration

The following entries must be appended to


# added manually
dhcpd_flags="athn0 dc0" # for normal use: ""
apmd_flags=NO           # for normal use: ""
ifstated_flags=""       # for normal use: ""

Remember to check /var/log/* if anything goes wrong.

Step 3 - bandwidth problems after configuration

At this point, things were working for me. Connecting to smyslov with fischer, I could ping out, and DNS resolution was occurring.

From fischer, I performed another speed test. Unfortunately, it was much slower. 70% slower:

nosuchuser@fischer:~$ ministat -w 80 speedtest_before speedtest_after
x speedtest_before
+ speedtest_after
|   +    +                                                               xx      |
|   +    +       +                                                       xx      |
|+  +  +++ ++  + ++                                                 xx   xx   x x|
|   |____MA____|                                                      |__A__|    |
    N           Min           Max        Median           Avg        Stddev
x  15       2000160       2302560       2141280       2140704     73701.317
+  15        352716        763980        548352      564916.8     130480.22
Difference at 95.0% confidence
        -1.57579e+06 +/- 79242.8
        -73.6107% +/- 3.70172%
        (Student's t, pooled s = 105965)

With a single wget connection, I saw a download rate of ~100 KB/s with smyslov, when I saw ~500KB/s with the WRT54GL.

The problem was with how I had originally configured /etc/hostname.athn0. I had used the line

media OFDM54 mediaopt hostap

instead of

media autoselect
mode 11g
mediaopt hostap

The previous setting was throttling my connection. Interestingly enough, when I tested with multiple wget commands instead of one the connection seems to top out at ~430 KB/s, which is much more respectable and in line with what I saw with the WRT54GL.

smyslov’s load averages when at ~430 KB/s were also not bad to see:

0.13 0.18 0.09

I have since seen this configuration handle ~600 KB/s without problems.

Step 4 - Resolve DNS locally

I chose to install unbound so that I could have smyslov resolve DNS queries. With OpenBSD, it was as easy as calling pkg_add:

$ pkg_add -v unbound

I configured unbound mostly following the steps at, but I disabled remote control. Also, with my setup, I needed to specify ‘interface: ‘ to both work with both subnets and localhost.

Once configured, add to /etc/rc.local:

echo 'starting unbound '

Step 5 - Connection dropouts

smyslov was put into service between my LAN and the internet on Wed Oct 16, and unfortunately my persistent connections still disconnected regularly.

This time however much more so:



It also seemed to be getting worse.

Step 6 - Diagnosing a kernel PPPoE bug

OpenBSD’s ifconfig has the ability to enable excellent debugging information per interface simply by setting the debug flag on that interface:

ifconfig pppoe0 debug

I enabled the debug flag on the pppoe0 interface when the network was not noisy, and saw the following:

Nov  1 09:24:59 smyslov /bsd: pppoe0: lcp output <echo-req id=0xea len=8 00-00-00-00>
Nov  1 09:24:59 smyslov /bsd: pppoe0 (8864) state=3, session=0x3d22 output -> 00:90:1a:a3:7e:12, len=16
Nov  1 09:24:59 smyslov /bsd: pppoe0: lcp input(opened): <term-req id=0xbf len=4 00-00-00-...-00-00-00>
Nov  1 09:24:59 smyslov /bsd: pppoe0: lcp opened->stopping

According to RFC 1661, my computer was outputting bad echo-req packets:


      The Magic-Number field is four octets, and indicates a number
      which is very likely to be unique to one end of the link.  A
      Magic-Number of zero is illegal and MUST always be Nak'd, if it is
      not Rejected outright.

This is due to the following code in sys/net/if_spppsubr.c, found at github here:

else if (sp->pp_phase >= PHASE_AUTHENTICATE) {
        unsigned long nmagic = htonl (sp->lcp.magic);
        sp->lcp.echoid = ++sp->pp_seq;
        sppp_cp_send (sp, PPP_LCP, ECHO_REQ,
                sp->lcp.echoid, 4, &nmagic);

This is a bug because the Sun Blade 100 uses the UltraSparc IIe processor, which is a 64 bit big endian processor. From the Wikipedia article on endianness:

Big-endian systems are systems in which the most significant byte of the word is stored
in the smallest address given and the least significant byte is stored in the largest. In
contrast, little endian systems are those in which the least significant byte is stored in
the smallest address.

The unsigned long is 64 bits in this context, and when the address of &nmagic is passed, the most significant byte of the 64 bit unsigned long is seen, and the upper 4 most significant bytes are blank.

$ cat bigendian.c
#include <stdio.h>
#include <stdint.h>

int main()
        int j;
        uint64_t i = 0x11335577;
        uint8_t * p = &i;

        for( j = 0; j < sizeof(unsigned long); j++)
                printf("%02x ", p[j]);


        return 0;

$ cc -o be bigendian.c
bigendian.c: In function 'main':
bigendian.c:8: warning: initialization from incompatible pointer type
$ ./be
00 00 00 00 11 33 55 77

The fix is not to use an unsigned long to hold the magic number, but an int32_t:

$ diff /usr/src/sys/net/if_spppsubr.c.orig /usr/src/sys/net/if_spppsubr.c    
<                       unsigned long nmagic = htonl (sp->lcp.magic);
>                       int32_t nmagic = htonl (sp->lcp.magic);

Compiling a custom OpenBSD kernel to test this out was very straight forward.


1) Get the source (


# get -stable
cvs -d$CVSROOT checkout -rOPENBSD_5_3 -P src

# update -stable
cvs -d$CVSROOT up -rOPENBSD_5_3 -Pd

2) apply the patch

3) compile your architecture’s kernel (

# cd /usr/src/sys/arch/`machine`/conf
# config GENERIC
# cd ../compile/GENERIC/
# make clean && make
# make install

Reboot, and confirm that the new kernel has booted:

# uname -a
OpenBSD 5.3 GENERIC#0 sparc64


OpenBSD 5.3-stable (GENERIC) #0: Tue Nov  5 19:03:41 EST 2013

I booted the new kernel on Tue Nov 05, and the disconnects stopped happening as frequently:

They’re still happening, but not nearly as frequently, and as far as I can tell not because of the incorrect echo-req packets.

Note: from

The other architectures that OpenBSD supports have benefited because some kinds of bugs are exposed more often by the 64-bit big endian nature of UltraSPARC.

Step 7 - reporting the bug(s)

All 3 major BSDs use the same general if_spppsubr.c structure. OpenBSD uses unsigned long (bug), FreeBSD uses long (bug), but NetBSD uses int32_t. I expect NetBSD has been run on more architectures more frequently, and would have seen this endianness bug earlier because of this.

I’m in the process of reporting these bugs to both OpenBSD and FreeBSD.

Step 8 - conclusion

smyslov is still performing with no problems, but the main problem is that the box is quite noisy and requires about 40 watts to run. The main appeal of using the Sun Blade 100 was just to see if I could get it working, and to try using OpenBSD for the first time. I can see getting new hardware in the future that is quiet and uses far less power to run, but not yet - I’d like to run the Sun Blade 100 a little more.

Despite the bug, I had a very good experience with OpenBSD. A little bit of time to configure, but once configured it seems rock solid. It was also refreshing that it was so straight forward to configure - no surprises anywhere except when it was my fault. Debugging with tcpdump -i pppoe0 and ifconfig pppoe0 debug was a big help in diagnosing the endianness problem.

Additionally, like my previous experience with FreeBSD, OpenBSD custom kernel development, building, and booting was easy. I’ve tried before to compile and boot my own Linux kernel to no avail. With [Free/Open]BSD, it was as easy as make && make install && reboot. It’s probably as easy with Linux but I’ve never had that experience.