Dear lazyyweb,
Has anyone got a way for postfix to use a transport map such that it sends email to the given MX host for some specified domains then default to a smarthost for the remainder?
The logic would be:
IF domain in (‘example.net’, ‘foo.bar’,etc etc) THEN use the relevant MX host ELSE send to smarthost.isp.net
i can use a transport map to send from specific domains to specific hosts, eg @domain.com goes to otherisp.net mailserver but not to just use domain.com’s MX hosts
I hate spam, though I get an awful lot of it. About 1/3 of my email is spam, though on a bad day the ratio can be reversed. If you want to see just how much spam I get, I’ve used to have a nice graph of spam. To get rid of it I use a lot of filters. One of these is the Postfix Body Checks feature. This feature allows you to match lines in the body of the email and reject them at the server. I use Perl Compatible Regular Expression (PCRE) matching for the lines.
Recently though, I noticed a lot of spam, usually about Viagra, that was passing through my spam traps. I noticed the emails all talk about a small set of webservers, so I’ll just filter on the urls. It didn’t work.
SpamAssassin in the end gave me the hint.
X-Spam-Status: No, hits=0.0 required=5.0
tests=BASE64_ENC_TEXT,EMAIL_ATTRIBUTION,HTML_60_70,
HTML_IMAGE_ONLY_04,MIME_HTML_ONLY,PENIS_ENLARGE,REMOVE_PAGE
version=2.53
It was base64 encoded email! That’s why my simple PCRE text matches would not work. So I needed to use something else.
This page is about how to filter on base64 text that appears in emails. I have used examples of PCRE and postfix but you can use this anywhere else, with the appropriate adjustments of where the files go and their syntax.
A standard filter line in a postfix body_check file looks something like this:
// REJECT
This is the old iframe hack that some spammers use to sneak URLs into your email.They are nice and clear and we just reject them. All we have to do now is change the stuff between the “toothpicks” // to what we want.
Here’s an example spam I got today, its offering the usual garbage this shonks usually offer. Remember if they don’t advertise ethically, it is often a sign of their entire operation.
In this case, I’ve decided I cannot be bothered getting any emails that advertise stuff on www.sellthrunet.net, I get enough junk already and it’s probably a front for spammers anyway, so I’ll filter on that domain. You need to make the string reasonably long as you are effectively cutting off parts of it.
Debian systems have this program called mimencode, some of you might have mmencode, which is part of the Metamail package. This does the base64 encoding for you.
So all you need to do is take the string you want filtered on, put it into mimencode and then put the resulting string into the postfix configuration. You need to do this three times, deleting a character at the front each time because base64 is done by cutting the strings up into groups of three characters each and you don’t know in advance if the your string is going to start at position 1,2 or 3.
gonzo$ echo -n "http://www.sellthrunet.net/" | mimencode HR0cDovL3d3dy5zZWxsdGhydW5ldC5uZXQv gonzo$ echo -n "ttp://www.sellthrunet.net/" | mimencode dHRwOi8vd3d3LnNlbGx0aHJ1bmV0Lm5ldC8= gonzo$ echo -n "tp://www.sellthrunet.net/" | mimencode dHA6Ly93d3cuc2VsbHRocnVuZXQubmV0Lw==
Next you need to remove part of the encoded string at the end. Remember that 3 characters are encoded into 4 symbols. So character one contributes to symbol 1 and 2, two to 2 and 3 and three to 3 and 4. The = means the string was not a multiple of 3 and it needs padding. If the encoded string has no =, you can use it as-is, otherwise remove all = plus one more character at the end of the string. Remember that you are cutting off up to two characters from your regular expression from both ends so be careful it is still meaningful. The last string for example is only matching “tp://www.sellthrunet.ne” which still looks ok.
Finally, you can join the strings using the regular expression “or” symbol. Also be careful to escape any strings that use special regular expression characters. Base64 can have plus ‘+’ and slash ‘/’ which need
escaping with a backslash .
(HR0cDovL3d3dy5zZWxsdGhydW5ldC5uZXQv|dHRwOi8vd3d3LnNlbGx0aHJ1bmV0Lm5ldC|dHA6Ly93 d3cuc2VsbHRocnVuZXQubmV0L)
I have a bypass line in my setup so usually any lines that are base64 encoded are bypassed, so if you have the same thing make sure this line goes before your bypass line or it will never match. We also need to tell postfix to use case sensitive matching because it is base64 hash we are matching and not the real string itself, so we use the i flag after the last slash . The relevant lines in the body_checks file are now:
#
# sellthrunet.net
/(HR0cDovL3d3dy5zZWxsdGhydW5ldC5uZXQv|dHRwOi8vd3d3LnNlbGx0aHJ1bmV0Lm5ldC|dHA6Ly93d3cuc2VsbHRocnVuZXQubmV0L)/i REJECT Spamvertised website
# don't bother checking each line of attachments
/^[0-9a-z+/=]{60,}s*$/ OK
To test it, I use pcregrep and mimencode again, on the mail file. This will show in clear text the spamming line and gives you an idea that it should work.
$ pcregrep 'dHRwOi8vd3d3LnNlbGx0aHJ1bmV0Lm5ldC' /var/mail/csmall | mimencode -u http://www.sellthrunet.net/pek/m2b.php?man=ki921"><l;im//www.sellthrunet.net/pek/m2b.php?man=ki
There are times when you need to have users authenticate their SMTP sessions. Perhaps you have roaming users and you don’t want to be an open relay, but you cannot predict where these users are. You need a way for them to say to your SMTP server “hey I belong here, let me send email”.
One way to do is is using SMTP Authentication. The user’s username and password are sent to the SMTP server. The server then checks the pair is correct and lets the user then send mail (or not if they are incorrect). SMTP Authentication is defined in RFC 2554.
Postfix has a method of authentication, but it is tied up with SASL. For file-based authentication you just create a special password database. However for other types you cannot simply make a LDAP or MySQL table and be done with it. You can either use SASL natively or do it the way I have implemented it here where Postfix uses SASL which uses PAM which uses MySQL; around-about way but it does work. There is some sporadic documentation about this around The Internet, but I wrote this up in the hope you find it useful and so I don’t have to remember it or relearn it all over again.
You might also be able to adapt this method to use other sorts of PAM authentication. For example I’m pretty sure this method with a little adaption would also work for LDAP authentication. Obviously you could
use other databases other than MySQL, its just what I was using here.
The following Debian packages are required to get this all working. I’m using Debian Sarge here but for the most part it should work for other versions and dists with some small changes. Some other packages will be needed, but will be pulled in as dependencies.
You have to make sure that either one or both of the authentication modules packages are installed. If you don’t and you setup Postfix to use SASL (see below) then the stupid process will be throttled. For older distributions you may need the libsasl (no 2) packages.
Postfix runs the smtpd daemon in a chrooted environment, usually something like */var/spool/postfix*. That means that as far as the smtpd process is concerned you have nothing above that point. MySQL has a socket sitting in another directory, something like */var/run/mysql/mysqld.sock*. The problem is that the socket sits in an area that smtpd believes doesn’t exist and cannot get to anyway because of the chroot.
To get around this problem, you have 3 options:
1. Stop smtpd from running into a chroot.
2. Move the mysql socket into the chroot.
3. Don’t use the mysql socket, use a TCP socket instead.
The last two are reasonably simple, possibly the third is the best option (you can make mysqld listen only to the loopback interface). Look at the MySQL documentation about how to move sockets or make it listen on its TCP port.
This had me going for a long, long time. To change this, edit /etc/postfix/master.cf and change the following line:
smtp inet n - n - - smtpd
The second ‘n’ means it is not chrooted. There may be a way of running smtpd in a chroot with the SASL and MySQL authentication but I’m not sure how.
The following lines are added to /etc/postfix/main.cf
smtpd_sasl_auth_enable = yes smtpd_sasl_local_domain = myserver broken_sasl_auth_clients = yes smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, reject
So far the postfix server knows it has to use SASL if it gets an authentication request. The default way for SASL to work out if you are authenticated is for it to examine a Berkley DB file called /etc/sasldb2. You can add and change users using the saslpasswd2 program.
The problem here is if you run smtpd in a chroot environment then it will not find the sasldb file. If you try to authenticate postfix will give an error “warning: SASL authentication problem: unable to open Berkeley db /etc/sasldb2: No such file or directory”. The problem here is that you have a /etc/sasldb2 file, but postfix is looking for a /var/spool/postfix/etc/sasldb2 file.
The two solutions for this problem are to either not run postfix in a chroot environment (see a previous section on how to stop it) or get that sasldb2 file into the correct directory. You can put it right by copying it. You will also need to make sure the user that smtpd runs as can read the file.
Debian users can automatically get this file updated by editing /etc/init.d/postfix. Around line 43 there is a list of files that are copied from their real directories into the chroot. Change the line so it looks like:
FILES="etc/localtime etc/services etc/resolv.conf etc/hosts
etc/nsswitch.conf etc/sasldb2"
Now when postfix is restarted you have the new sasldb2 ready to go.
If you are doing file-based authentication then you are done, drop down to the Testing section.
For MySQL authentication, the next step is to get SASL to ask PAM to authenticate the user. There’s some confusion because the location of this file has moved around. On my system with the versions of the packages given above, it is found at /etc/postfix/sasl/smtpd.conf but it also has been found in /usr/local/lib/sasl/smtpd.conf and /usr/lib/sasl/smtp.conf. The file is real simple one-liner:
pwcheck_method: pam
That’s it for SASL, it will then use standard PAM as we all know and love for authenticating.
The PAM setup is pretty standard. All you need to know is the PAM service is called smtp, so you need to create a file /etc/pam.d/smtp. SASL only uses the authentication management group.
It might be useful to test how things are going so far. To do this, and only for testing, you can use the pam_permit module. This module permits anything you send, so its useful for testing or for some strange circumstances, but shouldn’t be used in a production environment. The file /etc/pam.d/smtp would then look like:
auth required pam_permit.so
If you are going to run it with MySQL, use a configuration similar to that shown below. The configuration is similar to a user doing the following:
server$ mysql -u postfix -psecret postfixdb mysql> SELECT id FROM users WHERE id='givenusername' AND password='givenpassword';
auth required pam_mysql.so user=postfix passwd=secret db=postfixdb table=users usercolumn=id passwdcolumn=password crypt=0
The table users has two columns. The first is called id and has the username, the second is password it has the unencrypted password in it. A select is made checking both username and password. If there is a single row returned, authentication is successful.
I use the plain authentication method for testing. To do this you need to convert the username and password into a base64 encoded string. For example, if you have username user and password pass, you would type:
server$ printf 'useruserpass' | mimencode dXNlcgB1c2VyAHBhc3M=
So the string is the username and password joined together with between them. The username is needed twice. To test it, telnet to the SMTP port of your server and type the auth commands.
server$ telnet mail.my.server 25 Trying 10.1.2.3 Connected to 10.1.2.3. Escape character is '^]'. 220 mail.my.server ESMTP Postfix EHLO blah 250-mail.my.server 250-PIPELINING 250-SIZE 10240000 250-VRFY 250-ETRN 250-AUTH LOGIN PLAIN CRAM-MD5 DIGEST-MD5 250-AUTH=LOGIN PLAIN CRAM-MD5 DIGEST-MD5 250-XVERP 250 8BITMIME auth plain dXNlcgB1c2VyAHBhc3M= 235 Authentication successful
I’ve used a EHLO instead of the normal HELO as this is an extended hello, so the server gives you a list of things it can do. Notice that there are two AUTH lines, this is due to the broken_sasl_auth_clients line in /etc/postfix/main.cf.
You may have different authentication modules, it depends on what packages you have installed.
The important thing is the server’s response to your commands is 235 Authentication successful. This means that it recognizes the username and password. If it doesn’t, it returns a 535 Error: authentication failed. If you get a failed message, check the mail logs. The logs should tell you why the authentication failed.
Instead of using the plain authentication, you might want to use the LOGIN method. Once again mimencode is used to get the base64 encoding:
server$ printf 'user' | mimencode dXNlcg== server$ printf 'pass' | mimencode cGFzcw==
You now have the two base64 encoded strings, to test this method is very similar to the PLAIN method.
server$ telnet 10.1.2.3 25 Trying 10.1.2.3... Connected to 10.1.2.3. Escape character is '^]'. 220 my.mail.server ESMTP Postfix EHLO blah 250-my.mail.server 250-PIPELINING 250-SIZE 10240000 250-VRFY 250-ETRN 250-AUTH LOGIN PLAIN CRAM-MD5 DIGEST-MD5 250-AUTH=LOGIN PLAIN CRAM-MD5 DIGEST-MD5 250-XVERP 250 8BITMIME auth login 334 VXNlcm5hbWU6 dXNlcg== 334 UGFzc3dvcmQ6 cGFzcw== 235 Authentication successful
You might wonder what that strange text is after the 334 numbers. Once again mimencode can help. It’s a base64 encoding of the response from the mail server.
server$ printf 'VXNlcm5hbWU6' | mimencode -u ; echo Username: server$ printf 'UGFzc3dvcmQ6' | mimencode -u ; echo Password:
So the mail server is asking for a username and password, in base64. I don’t know why they bother to do this as it doesn’t make it that much more secure but at least you now know what it is.
OK, so you have you server setup that can do authentication, but now you want your laptop that is running Postfix to relay all email through your server. This section describes the client setup.
Setting up Postfix is pretty simple. Tell Postfix to send all email to your mail server and enable SASL. The file /etc/postfix/main.cf requires the following lines:
relayhost = mail.example.net smtp_sasl_auth_enable = yes smtp_sasl_password_maps = hash:/etc/postfix/sasl/passwd smtp_sasl_security_options =
The configuration is telling postfix to send all email to mail.example.net, use SASL authentication and that the passwords are found in a particular file. Remember for outgoing mail Postfix uses smtp while incoming
uses smtpd. As the client sends email the configuration lines have the “d less” smtp_ keywords.
The format of the client password file is simple, especially if you have written hash tables for Postfix before. The key is the remote server and the value is the username and password to use for that server separated by a colon.
mail.example.net myuser:secpasswd