This post details the motivation and methods used to move operand.ca’s backing VPS from Linode to Ramnode.
Table of contents:
For a while before this post, I had been looking at other VPS providers to back operand.ca. I’ve been a Linode customer since 2012 and have no complaints, but I feel like I am paying too much for too large of a VPS instance given what operand.ca is: this static site and a mail server. I was also looking to move to OpenBSD for my VPS OS.
After searching for VPS providers with OpenBSD support, Ramnode stuck out: good reviews, and good price point.
35$ a year buys one a very small VPS at one of four datacenter locations:
256 MB RAM
1 Core CPU:
kern.ccpu=1948
hw.model=QEMU Virtual CPU version (cpu64-rhel6)
hw.ncpu=1
hw.cpuspeed=2600
hw.ncpufound=1
machdep.cpuvendor=GenuineIntel
machdep.cpuid=1747
machdep.cpufeature=127663101
30 GB SSD-cached storage
1 TB bandwidth
I figured that those specifications should be plenty to replace the bulkier Linode.
In Ramnode’s SolusVM control panel, there was no option to “reinstall” OpenBSD 5.6, so I had to mount the OpenBSD 5.6 AMD64 ISO and install using the HTML5 VNC window.
I wanted to have a strong root password and use full disc encryption, but both of those required long random passwords and pasting text into the HTML5 VNC window was troublesome. It wouldn’t work reliably for me until I used xdotool:
xdotool search VNC windowactivate type "<long random passphrase>"
xdotool search VNC windowactivate key Return
and after that it worked - I could reliably paste long passphrases into the HTML5 VNC window.
Using full disc encryption with OpenBSD 5.6 is as simple as:
Partitioning the root disc into one large partition as a target later for the encryption:
# fdisk -iy wd0
# disklabel -E wd0
> a b
offset: [64]
size: [62910476] 1g
Rounding size to cylinder (16065 sectors): 2104451
FS type: [swap]
> a a
offset: [2104515]
size: 60806025
FS type: [4.2BSD] RAID
> w
> q
Encrypting that large partition using bioctl:
# bioctl -c C -l /dev/wd0a softraid0
sd0 at scsibus1 targ 1 lun 0: <OPENBSD, SR CRYPTO, 005> SCSI2 0/direct fixed
sd0: 29690 MB, 512 bytes/sector, 60805497 sectors
softraid0: CRYPTO volume attached as sd0
sd0 (“CRYPTO volume attached as”) instead of wd0 as the install target.After that, reboot, enter your passphrase with xdotool, and type boot to boot into OpenBSD.
I then did the normal new system things:
pfhttpd setup was very straight forward:
/etc/httpd.conf:
prefork 2
server "default" {
listen on localhost port 8080
root "/htdocs/default"
}
/etc/rc.conf.local:
httpd_flags=""
Note that the root directory in this case is /var/www/htdocs/default.
varnish was a pkg_add away:
/etc/varnish/default.vcl:
backend default {
.host = "127.0.0.1";
.port = "8080";
}
/etc/rc.local:
/etc/rc.d/varnishd start
Around the time I was looking to switch to Ramnode, I also discovered that the base qmail installation that I had been using for a long time as my mail server was bouncing spam messages:
root@li129-126:~# /var/qmail/bin/qmail-qread
13 Nov 2014 22:17:34 GMT #157323 2087 <>
remote jaxqsvhyfbmyg@hillary4president.org
13 Nov 2014 14:53:25 GMT #157303 1402 <>
remote vfaoktx@inanities.com
14 Nov 2014 00:42:15 GMT #157328 2071 <>
remote pfsxidki@javierbaeza.com
15 Nov 2014 00:14:32 GMT #157191 1718 <>
remote blprmbqybwsyq@htclub.org
13 Nov 2014 21:49:22 GMT #157317 1914 <>
remote hexudwu@fanal.info
Here is an example of this, which I discovered is called backscatter. spamtrap@operand.ca is a non existent user, so sending mail there creates a bounce message:
$ telnet operand.ca 25
Trying 69.164.214.126...
Connected to operand.ca.
Escape character is '^]'.
220 operand.ca ESMTP
HELO operand.ca
250 operand.ca
MAIL FROM: <james.w.macmahon@gmail.com>
250 ok
RCPT to: <spamtrap@operand.ca>
250 ok
DATA
354 go ahead
From: james.w.macmahon@gmail.com
To: spamtrap@operand.ca
Subject: fake message
This is a fake message from telnet. It should bounce.
.
250 ok 1416434891 qp 17522
quit
221 operand.ca
Connection closed by foreign host.
Note: more on the RCPT TO: later.
The resulting bounce is seen in the qmail logs:
new msg 157184
info msg 157184: bytes 316 from <james.w.macmahon@gmail.com> qp 17522 uid 110
starting delivery 216: msg 157184 to local spamtrap@operand.ca
status: local 1/10 remote 0/20
delivery 216: failure: Sorry,_no_mailbox_here_by_that_name._(#5.1.1)/
status: local 0/10 remote 0/20
bounce msg 157184 qp 17525
end msg 157184
new msg 157201
info msg 157201: bytes 858 from <> qp 17525 uid 111
starting delivery 217: msg 157201 to remote james.w.macmahon@gmail.com
status: local 0/10 remote 1/20
delivery 217: success: 64.233.171.26_accepted_message./Remote_host_said:_250_2.0.0_OK_1416434892_v2si638858qaf.48_-_gsmtp/
status: local 0/10 remote 0/20
end msg 157201
tcpserver: end 17521 status 0
tcpserver: status: 0/40
and on GMail:
Through reading about this, I came upon Chris Siebenmann’s blog post The minimum antispam features of a modern SMTP server, which unfortunately called out qmail as being bad at this:
2. reject mail to nonexistent users during the SMTP transaction.
Everyone should do #2, because otherwise your SMTP server is going to spam
innocent third parties when spammers forge their address into MAIL FROM and
send spam to nonexistent users on your machines. (Yes, I'm looking at you,
QMail; please wake up and join the 21st century.)
Wikipedia calls me out directly on their backscatter page :) :
Backscatter occurs because worms and spam messages often forge their sender
address, and mailservers configured by naive administrators send a bounce
message to this address.
There are plenty of patches for qmail which can deal with things like this, but I instead switched to OpenSMTPD, the base OpenBSD mail server. It rejected mail to non existent users by default:
$ telnet localhost 25
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
220 janssen.my.domain ESMTP OpenSMTPD
helo localhost
250 janssen.my.domain Hello localhost [127.0.0.1], pleased to meet you
mail from: <james.w.macmahon@gmail.com>
250 2.0.0: Ok
rcpt to: <spamtrap@operand.ca>
550 Invalid recipient
quit
221 2.0.0: Bye
Configuration for smtpd was also straight forward:
/etc/mail/smtpd.conf:
listen on all
table aliases db:/etc/mail/aliases.db
accept from any for domain "operand.ca" alias <aliases> deliver to maildir
accept for local alias <aliases> deliver to maildir
accept from local for any relay
where aliases.db was filled with wildcard addresses that I used with qmail (jwm-*), which do not work with OpenSMTPD. jwm+* however does work and this is the wildcard format I will have to use in the future. I found each jwm-* address using the following shell script in my Mail directory:
#!/bin/sh
find ./ -type f -print0 | \
xargs -0 grep To | \
grep "jwm\-" | \
sed -e 's/^.*\(jwm-[^@]*\)\@.*$/\1/g' | \
sort | \
uniq
I also enabled spamd. My configuration uses the base spamd.conf but with my own nospamd:
override:\
:white:\
:method=file:\
:file=/etc/mail/nospamd:
This file is populated with the ips and netblocks from SPF results generated by querying web mailers like GMail and the domains of the mailing lists that I follow.
As for the RCPT to earlier: I’ve also set up spamtrap@operand.ca to catch email harvesting scripts: spamd has a GREYTRAPPING feature where any email sent to a greytrapped address automatically blacklists the sender:
spamdb -T -a 'spamtrap@operand.ca'
Originally I used info instead of spamtrap (as seen in the GMail image). I wanted to use spamtrap on this page instead so that if (when) email harvesting occurs it will get that address instead.
Just in case I didn’t make it clear: email harvesting scripts, please contact me at spamtrap@operand.ca, or spamtrap at operand.ca. :)
dovecot was also a pkg_add away. However, it didn’t work initially:
# /usr/local/sbin/dovecot
Warning: fd limit (ulimit -n) is lower than required under max. load (128 < 1000), because of default_client_limit
Jan 09 22:37:25 master: Info: Dovecot v2.2.10 starting up for imap, pop3, lmtp
Jan 09 22:37:25 master: Error: service(imap): pipe() failed: Too many open files
Jan 09 22:37:25 master: Error: service(imap-urlauth-worker): pipe() failed: Too many open files
Jan 09 22:37:25 master: Error: service(imap-urlauth): pipe() failed: Too many open files
As per http://comments.gmane.org/gmane.os.openbsd.misc/207288:
/etc/login.conf:
# Dovecot
dovecot:\
:openfiles-cur=2048:\
:openfiles-max=4096:\
:tc=daemon:
After changing login.conf, /etc/rc.d/dovecot start worked.
My /etc/dovecot/dovecot.conf is fairly minimal:
mail_location = maildir:~/Maildir
log_path = /var/log/dovecot.log
info_log_path = /var/log/dovecot_info.log
login_greeting = Ready.
disable_plaintext_auth = no
auth_username_chars = abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@
ssl = no
auth_mechanisms = plain
auth_verbose = yes
auth_debug = yes
mail_debug = yes
first_valid_uid = 1000
passdb {
driver = passwd-file
args = scheme=BLF-CRYPT username_format=%u /etc/dovecot/users
}
userdb {
driver = passwd-file
args = username_format=%u /etc/dovecot/users
}
syslog_facility = dovecot
listen = 127.0.0.1
protocols = imap
service imap-login {
inet_listener imap {
port = <some high random port>
}
service_count = 1
}
Dovecot never faces outward - connections to dovecot are over ssh - but I still use strong passwords in case pf fails.
Switching operand.ca from the Linode to the Ramnode ultimately meant changing over the DNS and reverse DNS entries. I did this on January 9th and hopefully haven’t missed anything…
I did change the SPF record for operand.ca from
v=spf1 +a -all
to
v=spf1 +a ~all
because -all does not allow your mail to be relayed at all, where ~all does.
All that remains now is to move all the data I care about off the Linode and decomission it.