My Most Important Client On Planet Earth
Normally I sanitize articles to keep things anonymous. Not
this time - I'm going to call everyone out by name.
My most important client in this world is Christi, my wife. When she
brings me a problem, it is usually an important problem that needs to be
solved quickly and efficiently. She is also the IT director and a
security lead for her employer - so she knows her way around technology.
We belong to a wholesale buying service called DirectBuy and have for
> 25 years.
She has been trying to get into her account for a few days - password
reset emails were not arriving and the DirectBuy support people were not
able to fix it. Each time something would be done and she'd get a 'Check
again in 24-48 hours'.
So while she is on hold with DirectBuy technical support yet again
she asked me if I would take a look at it.
Warning: Cool and geeky technical content follows!
If you own a propeller beanie, now is the time to go get it.
I'll speak in both geeky and less-geeky so hopefully I won't
lose anyone along the way.
If you've read any of my other articles, I probably call out
Wireshark in at least 33% of them. Briefly, Wireshark is a tool
that lets anyone capture and look at the data as it travels over
a network. Like any tool, it can be used for good or evil. Since
I run my own mail server, I can run any tool I want on it - and
Wireshark was installed probably the day after I set the server
I fire up Wireshark, start a capture, and tell her to try the
password reset email link again.
Filtering a big data capture is an art form itself - there are many
paths that lead to the answer. Rather than enumerate them all I'll show
you the first thing I tried along with my reasoning and show the
I needed to find the specific password reset email that may or may
not have been sent. To do this, I first setup a filter to look for the
word 'directbuy' in any data packet:
All pictures are thumbnails - click on it to see a full size version.
The filter is the first box outlined in red with a green
background - that looks for the word directbuy in any packet. The second
red box shows protocol of SMTP, which stands for Simple Mail Transfer
Protocol - this is the way most email servers talk to each other over
the Internet. I've got a packet that deals with directbuy, now I want to
see the whole conversation the two servers had with each other... so
right click on one of the packets and select "Follow TCP Stream", and
you'll get this:
The blue represents my server - the receiving email server - and the
red is the sender - directbuy's server. If you look at the last line on
the screen, you can see they are talking about a password reset email.
There is a lot of gobbledeegook in this message that represent the
two servers discovering each other, what they are both capable of,
additional information that exists behind the scenes. But this is how
most all mail looks when you peek behind the curtain and watch the
wizards moving all the levers and flipping all the switches.
Right after the last blue line, there is a red line that says "BDAT
8287" - this says the sender is sending a big blob of binary data, AKA:
Chunking, and the server shouldn't be looking at each line-by-line just
yet. So most of the rest of the message - at least the next 8K or so of
data will be stuff that composes the message.
Incidently, I was able to grab this bit of the captured data, send it
to Christi, and she was able to reset her password as this was indeed
the password reset email along with the link that let her get back into
her account. So the main problem was solved
worked around - for us at least. But it is still a problem for many
Scrolling all the way to the bottom of the mail:
The magic line here comes from my server:
"550 5.7.1 Sender ID (PRA) Not Permitted"
Sender ID and SPF (Sender Policy Framerwork): What it is and why
you want it
Remember this: The internet was created by a bunch of techie
folks to try to communicate data between two points and thus
facilitate sharing information. Compare it to our roads which
were designed to allow easy transportation between destinations.
The Internet wasn't originally designed to be a secure and safe
but a way to transport data between locations. The roads in your
town weren't designed to only allow good people with good
intentions to drive on them - evil people can also drive on
those same roads.
Everything we've done to make the internet safer has been done after
the fact as a bolt-on addition.
Sender ID and SPF are two of those bolt-ons designed for one specific
purpose - to prevent someone from impersonating another via email - we
call that "Spoofing"
Here is an example of a spoofed message:
Now - does anyone actually believe that Bill Gates sent me this email
message? I hope not. Spoofing email is trivially easy - I can show a 7
year old that knows how to spell how to spoof an email in about 10
If you've configured your server to use Sender ID, then your server
will check and if the server that is sending the email isn't on a list
of servers that are authorized to send mail for that domain, then the
mail will either be marked as spam or outright rejected. If the sending
domain hasn't setup their Sender ID credentials called the SPF record,
then the domain is 100% spoofable ... so anyone can impersonate that
company and you might never know that you aren't really emailing with
your bank. SPF stands for "Sender
Policy Framework" which you can read more about at the link if you
are curious. A shorter and less detailed but friendlier article can be
(Nit: I've used Sender ID and SPF kind of interchangably, but they
really aren't the same... but they are VERY similar in how they operate
and their syntax. You can read about those nits I just picked at
The server I used to send the mail that looks like it came from Bill
Gates wasn't setup to use SPF ... so the mail could be spoofed. Had it
checked the SPF records, it would have been either rejected ("-all") or
accepted & marked suspicious ("~all") the mail - more on that later.
Please explain to me: How this is a problem with DirectBuy and
not with your server?
Yes, my server rejected the message.
Should it have rejected it or should it have let the message through?
Answering that is going to be geeker than most want to read ... so
proceed only if you have your propeller hat on and spinning.
Lets look at directbuy's SPF records:
I'll copy the two relevant records here:
So line 1 says the only servers that can send mail that looks like it
comes from directbuy will be at the SPF records looked up at
spf.protection.outlook.com, aspmx.sailthru.com, and
cust-spf.exacttarget.com. In other words, we won't tell you which
servers directly but refer you to other lists of other servers to decide
if things are good or not good. So we (or the server) has to check the
SPF records starting with spf.protection.outlook.com and if that has
includes go through those and if those have includes go a level deeper.
In the computer science world we call this "recursion".
Here are the spf records at spf.protection.outlook.com:
Now he is giving us some IP addresses we can check against ... and
another list (spfa.protection.outlook.com) to look at as well.
Oh look! spfa wants us to include spfb.protection.outlook.com as
well! (get the idea? This recursive stuff can go on and on...)
Rather than recurse down that rabbit hole, I'm going to skip a few
steps and look for the 'sailthru.com' which is what the server that was
trying to send the password reset email claimed to be - see the 2nd
print screen's 2nd line from the top of the article. Here is the SPF
record for aspmx.salilthru.com:
"v=spf1 ip4:22.214.171.124/27 ip4:126.96.36.199/26
ip4:188.8.131.52/24 ip4:184.108.40.206/24 ip4:220.127.116.11/24
ip4:18.104.22.168/24 ip4:22.214.171.124/24 ip4:126.96.36.199/23
DirectBuy's password reset email is coming in from 188.8.131.52 -
look at the "source" column on the first print screen - and if you look
at the line above that is the 3rd IP address listed. So DirectBuy's
email server is allowed to send in email and claim it is from the domain
But wait - we aren't done yet. SPF was the first record type I
checked - DirectBuy also specified a SenderID record, that is the one
that looked like #2 - repeating both lines here:
Line 2 has us checking spf.protection.outlook.com, is missing both
aspmx.sailthru.com and spf.exacttarget.com, and adds ucctradingcorp.com
and cust-senderid.exacttarget.com. Also notice the suffix "-all" - this
says to hard fail if not on the list vs. "~all" that says to mark it as
suspicious and pass it into the server anyway.
Now lets traverse the spf2.0/pra path! Here are both:
No SPF or SenderID records at ucctradingcorp.com, so nothing to see
there, folks! If you are going to tell systems how to know you are
authentic, you need to be sure all the systems that can claim they are
from you know how to prove their authenticity.
So why is it listed as a place to check for SPF? A great question to
ask whatever tech person set this up incorrectly.
cust-senderid.exacttarget.com has a nice list:
cust-senderid.exacttarget.com. 300 IN TXT
"spf2.0/pra ip4:184.108.40.206/24 ip4:220.127.116.11/23 ip4:18.104.22.168/20
ip4:22.214.171.124/20 ip4:126.96.36.199/21 ip 4:188.8.131.52/24 "
"ip4:184.108.40.206/27 ip4:220.127.116.11/24 ip4:18.104.22.168/28
ip4:22.214.171.124/20 ip4:126.96.36.199/20 ip4:188.8.131.52/20
ip4:184.108.40.206/16 ip4:220.127.116.11/24 ip4:18.104.22.168/24 -all"
Look carefully - do you see 22.214.171.124 covered? Anything even
close? In case you aren't familiar with subnet masks and their notation,
I'll just tell you the answer: "No"
And the "-all" says to hard fail if someone is sending and they
aren't on the list - so this message must be from some evil spoofing
phishing spamming group that is up to no good, so you should just reject
Which is exactly what my server did - what DirectBuy's configuration
told my server to do - rejected the message. Because they told it to.
This is a very common problem - some tech folks configure
things based on wizards and how-to documents sometimes putting
together multiple sources without understanding how each one
works and you end up with interoperability issues. Since some or
a lot of the systems appear to work, they will blame anything
that doesn't work on the end user, or their email provider, or
their interent service provider, or their computer's antivirus
software, or anything else that lets them close the trouble
ticket and hang up the phone call. I wasn't on the call but I
can here one of the tech people trying a password reset, seeing
it come in, and saying to a poor helpless user "Well it worked
for me - I just tried it - so the problem must be on your end."
Diving Deeper Down the Rabbit Hole!
I'm totally done with discussing what was wrong - now I'm
curious just how Exchange 2010 processes the SPF & Sender ID
stuff - and I don't have to setup anything new, I already have
the network trace to look through.
Because this is how to learn things - get in, play around, see how
things operate and try to understand how it is thinking. If you know how
things work to a deep enough level, when something goes completely wonky
you'll have an idea of what could have gone wrong to cause those
Tweaking my filter - I could show just the DNS queries with the
filter 'dns' or the exact conversation between our mail servers by
filtering on tcp stream #2 (tcp.stream eq 2) ...so combining them I'll
get visibility into how Exchange processes both!
For the record, my filter is now:
dns || (tcp.stream eq 2)
There will be some redaction in these screen shots to eliminate any
license / proprietary information.
229-231 The connection is first established between the two mail
servers - the one doing the password reset mail
(ny1mta-138.sailthru.com) and mine. Just prior and after (redacted lines
218, 237, and 238) are my server checking the Trend Micro "Evil Server
Blocklist" to see if the server is listed. Also early on my server is
checking the barracuda block list, spamcop, and spamhaus - if the server
is listed on ANY of these block lists my server will likely just hang up
the phone kind of like when you hear a robocaller is calling your phone.
This happens VERY early in the conversation.
The two servers are talking normally to each other so far - this is
early in the conversation.
304: During the conversation, Exchange sends a query for any TXT
and in 305 we get back all the TXT records - I only expanded the
interesting ones, but there are 5 txt records:
Exchange now goes through the SPF2.0/pra txt record - apparently this
is more important to him than the spf1 record is - somthing I did not
know until just this moment, which is why I do this kind of analysis:
343: Query for the list for spf.proection.outlook.com
348: Query for
spfb.proteciton.outlook.com (these are all due to the first include:
351: Query for cust-senderid-exacttarget.com
353: Query for
355: Response for 353 saying "no txt records for
356: Response for 351 answering txt records for
and then the magic answer - none of those lists referenced the IP
address of the sending server 126.96.36.199, so in packet 594 Excange
sends back the error message "550 5.7.1 Sender ID (PRA) Not Permitted"
At this point Exchange must have decided that there is no point in
looking at the spf1 record or list of servers there - it already knows
based on SenderID that this is a fraudulent email so give the error and
tell them to go away.
After a short delay (~2 seconds):
1292: The sending server sees the error message and realizes there is
no point in continuing to try to send - he has been called out as a
fraud, so he sends the "QUIT" command.
1293: Exchange ACKs the quit
with a 221 2.0.0 Service closing transmission channel response
The FIN - ACK packet saying to the TCP layer "Disconnect this socket"
and "I heard you want to shut this off"
1295: The RST (Reset) and ACK
packet shutting the socket off and also saying "Shutting down now -
There you go - way more than you ever wanted to know about
mail and a small bit of how mail is currently sometimes
authenticated as coming from a legitimate server that is
authorized to send mail for a particular domain.
If and only if it is configured correctly.
An interesting quesiton which I can't answer because DirectBuy
doesn't have SenderID configured correctly is if Exchange would pass
SenderID and then check the spf1 record list or does it say that 'if
SenderID passes then don't bother checking the SPF record". In other
words, this onion might have more than one layer. I could test this if
they fix the first error and break their spf1 record in the same way. Oh
the things I would do with infinte time on my hands.
If you found this helpful, please send me a brief email -- one line will more
than do. If I see people need, want, and / or use this kind of information that
will encourage me to keep creating this kind of content. Whereas if I never hear
from anyone, then why bother?
If you are having problems and need someone to help diagnose them, I can
be reached at the Contact link on the top left menu bar.
You can email me at:
das (at-sign) dascomputerconsultants (dot) com
Copyright (C) 2019 DAS Computer Consultants, LTD. All rights