MariaDB + Debian

For MariaDB fans there are some changes coming in the current Debian testing/unstable, which will hit mainstream sometime this year (I guess).
A lot of things compiled against libmariadbclient.so.18 will start failing, most of which can be solved with a simple symlink except a few…

Opendmarc wil fail to compile:
opendkim: malloc.c:3757: _int_malloc: Assertion `(unsigned long) (size) >= (unsigned long) (nb)' failed.

Spamassassin will fail to run
spamd[27186]: at /usr/share/perl5/Mail/SpamAssassin/Conf/SQL.pm line 138.
spamd[27186]: spamd: service unavailable: Error fetching user preferences via SQL
spamd[3690]: config: failed to load user (dmarc@black-mail.nl) scores from SQL database: install_driver(mysql) failed: Attempt to reload DBD/mysql.pm aborted.
spamd[3690]: Compilation failed in require at (eval 1409) line 3, line 2.

The opendkim one is pretty vague and seemingly unhelpfull but a bit of digging will reveal the following:
Can't load '/usr/lib/x86_64-linux-gnu/perl5/5.24/auto/DBD/mysql/mysql.so' for module DBD::mysql: /usr/lib/x86_64-linux-gnu/perl5/5.24/auto/DBD/mysql/mysql.so: symbol mysql_options4, version libmariadbclient_18 not defined in file libmariadbclient.so.18 with link time reference at /usr/lib/x86_64-linux-gnu/perl/5.24/DynaLoader.pm line 187.
at /usr/sbin/opendmarc-import line 22.
Compilation failed in require at /usr/sbin/opendmarc-import line 22.

Hey, now we have a finger to point, Perl::DBD:
ldd /usr/lib/x86_64-linux-gnu/perl5/5.24/auto/DBD/mysql/mysql.so
/usr/lib/x86_64-linux-gnu/perl5/5.24/auto/DBD/mysql/mysql.so: /lib/libmariadbclient.so.18: no version information available (required by /usr/lib/x86_64-linux-gnu/perl5/5.24/auto/DBD/mysql/mysql.so)
linux-vdso.so.1 (0x00007ffdca9bc000)
libmariadbclient.so.18 => /lib/libmariadbclient.so.18 (0x00007faf4fe63000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007faf4fac5000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007faf4f8ab000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007faf4f68e000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007faf4f48a000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007faf4f184000)
libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007faf4ef18000)
libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007faf4ea85000)
/lib64/ld-linux-x86-64.so.2 (0x000056045c708000)

For some reason no symlinking will solve this, but no worries, simply get the source, unpack it and run:
perl Makefile.PL
make
make install
make clean

After this things should work smoothly, but keep in mind to keep the package up date date and remove the manually installed files should Debian fix this.

Effective dovecot+sieve extension filtering

After some recent conversations with a colleague about creating a simple shield that would stop some of the dangerous crypto/ransomeware I set about creating such a filter.

The premise was simple, move all mail containing predefined attachments under a certain size to a different folder. Sounds like a fine job for the sieve filter to me.

A quick search on the internet resulted in some basic ways to achieve this sort of behavior, however most were plain wrong in the execution of the regex statements. Which as it turns out should not even be used, the pigeonhole addon does all the required mime handling for us.

Problems with most solutions:

  • Regex statement per extension which creates an unreadable mess (and plain kills the server performance on larger emails)
  • Not limiting the search areas, this results in a lot of false positives (e.g. any of the extensions appearing AFTER the filename has been specified (within the encoded attachment) would trigger the rule)
  • Failing to correctly identify and target the different filename formats (filename=”test.zip”, filename=test.zip[\r\n] or filename=test.zip; and in extreme cases just a file= with no filename=)

The result code:

Now this doesn’t solve all problems presented with the current ransomeware (or other scams/virusses), but it does prevent a lot of the current exploits which were comming in in bulk.

Don’t forget to specify the folder in doveoct as a required folder:

Systemd @reboot cron replacement

Some of us enjoyed the old @reboot crontab entries but the recent upgrades to systemd have created some new challenges.
The most prominent being that a simple systemd service kills any spawned processes after it exits. (I know it is not supposed to be used like this, but for some simple services like mailgraph writing service files for everyone is just too cumbersome).

[Unit]
Description=Boot services

[Service]
Type=oneshot
RemainAfterExit=yes
KillMode=none
ExecStart=<boot-script>

December in the it department

While December for most people is a cheerful and happy month it is also has another side most commonly experienced by the IT department.

  • Spam rises notably
  • Hack attempts rise significantly (strangely the week after the Paris attacks they skyrocketed from French ip’s)
  • Server loads become more spastic, especially in retail

This year santa claus appears to be handing out free .biz extensions as a lot of spam (neatly formed and a great pain for the spam filters I might add) is arriving from freshly registered domains.

A quick example over the last 24 hours, for every ham e-mail there are 1.6 spam! (usually it’s below a 1:1 ratio)
spam 09-12-2015
Ham 09-12-2015

http/2 with Apache on Debian

One of the nice things to come out this year (at least on paper) is the the http/2 specification.
Most web servers still don’t support it as only the latest Apache (2.4.17) has inbuilt support for it, so only people who compile their own or use experimental distributions are lucky enough to get it.

Below is an easy patch to enable it on a stable Debian configuration, this server has been running part unstable for a long time but it does require some attention to detail.

Prerequisites

  • Up to date stable Debian installation
  • Some nerves / acceptable downtime
  • Configured ssl certificate, there is virtually no support for non https http/2

/etc/apt/sources.list add:

/etc/apt/preferences add:

Run:

/etc/apache2/apache2.conf add:

This method can also be used to install a more up to date version of other packages, but beware that packages in unstable are compiled against unstable libraries so this might not be a great idea to start upgrading everything you see.

Lag impact of SSL certificates

When purchasing (or getting/generating a free one) a SSL certificate the end-user performance is often overlooked (in most cases even unknown). I am not talking about selecting a bigger number of bits, always get the best available which has the support of the clients.

There is a second more sinister impact of securing data connections with a certificate, the chain length.

When getting a cheaper certificate the chain is usually longer, so there are more steps for the browser to perform, sure performing OCSP stapling mitigates this for most browsers.

But consider other connections which do not support this technology (smtp, imap, pop, mysql (including Mariadb), in fact most non https connections), this is were our problems start. For each connection most of these technologies have to re-verify the complete chain!

But after that the pain is far from over, please consider the party that has to respond to all these requests. After recently switching from StartSSL (paid, DV) to Comodo (DV) the average connection initialization time has seen a reduction of 40~50%!

This also has a measurable effect to the complete server, less open connections means more resources for doing useful stuff (it even has a considerable impact on the life of the IT staff, no more lag when moving emails about).

 

A pretty significant improvement for a simple upgrade, and it didn’t even require digging much deeper into my wallet.

Postfix, DKIM and random failed verifications

While using the ever so popular PHP to send outgoing mail through Postfix while signing them with a DKIM signature (opendkim, DKIMProxy or any other implementation) most of the time this simple setup seemed to work just fine.
However as our application grew some emails start to bounce or otherwise be rejected by a DKIM (body) verification error (Gmail, Hotmail etc all use DKIM verification nowadays, which is a good thing).

It took a solid hour to start figuring out what was going on, as it isn’t easy to see what broke the signature. A big clue comes from the fact that most emails verified just fine and most services specify which part fails.

The main culprit turned out to be an ancient limit on the maximum length of one line (separated by \r\n or sometimes denoted as <CR><LF>), as described in the Postfix manual: smtp line length limit

Normally this should not be a problem (one expects the DKIM filter/milter to be the final pass), except that Postfix chooses to enforce this policy after pickup or (non_)smtpd_milters…..
This problem is very easy to trigger, just insert a minified js or css into an email.

Simply adding the following code to your main.cf fixes this problem, so far no mail servers have mysteriously begun hating me.