 |
||||
         |
    |
|||
|
 |
|||
|
|
||||
|
My Most Important Client On Planet Earth Normally I sanitize articles to keep things anonymous. Not this time - I'm going to call everyone out by name. My most important client in this world is Christi, my wife. When she brings me a problem, it is usually an important problem that needs to be solved quickly and efficiently. She is also the IT director and a security lead for her employer - so she knows her way around technology. We belong to a wholesale buying service called DirectBuy and have for > 25 years. She has been trying to get into her account for a few days - password reset emails were not arriving and the DirectBuy support people were not able to fix it. Each time something would be done and she'd get a 'Check again in 24-48 hours'. So while she is on hold with DirectBuy technical support yet again she asked me if I would take a look at it. Warning: Cool and geeky technical content follows! If you own a propeller beanie, now is the time to go get it. I'll speak in both geeky and less-geeky so hopefully I won't lose anyone along the way. If you've read any of my other articles, I probably call out Wireshark in at least 33% of them. Briefly, Wireshark is a tool that lets anyone capture and look at the data as it travels over a network. Like any tool, it can be used for good or evil. Since I run my own mail server, I can run any tool I want on it - and Wireshark was installed probably the day after I set the server up. I fire up Wireshark, start a capture, and tell her to try the password reset email link again. Filtering a big data capture is an art form itself - there are many paths that lead to the answer. Rather than enumerate them all I'll show you the first thing I tried along with my reasoning and show the results. I needed to find the specific password reset email that may or may not have been sent. To do this, I first setup a filter to look for the word 'directbuy' in any data packet:
All pictures are thumbnails - click on it to see a full size version. The filter is the first box outlined in red with a green background - that looks for the word directbuy in any packet. The second red box shows protocol of SMTP, which stands for Simple Mail Transfer Protocol - this is the way most email servers talk to each other over the Internet. I've got a packet that deals with directbuy, now I want to see the whole conversation the two servers had with each other... so right click on one of the packets and select "Follow TCP Stream", and you'll get this:
The blue represents my server - the receiving email server - and the red is the sender - directbuy's server. If you look at the last line on the screen, you can see they are talking about a password reset email. There is a lot of gobbledeegook in this message that represent the two servers discovering each other, what they are both capable of, additional information that exists behind the scenes. But this is how most all mail looks when you peek behind the curtain and watch the wizards moving all the levers and flipping all the switches. Right after the last blue line, there is a red line that says "BDAT 8287" - this says the sender is sending a big blob of binary data, AKA: Chunking, and the server shouldn't be looking at each line-by-line just yet. So most of the rest of the message - at least the next 8K or so of data will be stuff that composes the message. Incidently, I was able to grab this bit of the captured data, send it to Christi, and she was able to reset her password as this was indeed the password reset email along with the link that let her get back into her account. So the main problem was solved worked around - for us at least. But it is still a problem for many others. Scrolling all the way to the bottom of the mail:
The magic line here comes from my server: "550 5.7.1 Sender ID (PRA) Not Permitted" Sender ID and SPF (Sender Policy Framerwork): What it is and why you want it Remember this: The internet was created by a bunch of techie folks to try to communicate data between two points and thus facilitate sharing information. Compare it to our roads which were designed to allow easy transportation between destinations. The Internet wasn't originally designed to be a secure and safe but a way to transport data between locations. The roads in your town weren't designed to only allow good people with good intentions to drive on them - evil people can also drive on those same roads. Everything we've done to make the internet safer has been done after the fact as a bolt-on addition. Sender ID and SPF are two of those bolt-ons designed for one specific purpose - to prevent someone from impersonating another via email - we call that "Spoofing" Here is an example of a spoofed message:
Now - does anyone actually believe that Bill Gates sent me this email message? I hope not. Spoofing email is trivially easy - I can show a 7 year old that knows how to spell how to spoof an email in about 10 minutes. If you've configured your server to use Sender ID, then your server will check and if the server that is sending the email isn't on a list of servers that are authorized to send mail for that domain, then the mail will either be marked as spam or outright rejected. If the sending domain hasn't setup their Sender ID credentials called the SPF record, then the domain is 100% spoofable ... so anyone can impersonate that company and you might never know that you aren't really emailing with your bank. SPF stands for "Sender Policy Framework" which you can read more about at the link if you are curious. A shorter and less detailed but friendlier article can be found here. (Nit: I've used Sender ID and SPF kind of interchangably, but they really aren't the same... but they are VERY similar in how they operate and their syntax. You can read about those nits I just picked at here.) The server I used to send the mail that looks like it came from Bill Gates wasn't setup to use SPF ... so the mail could be spoofed. Had it checked the SPF records, it would have been either rejected ("-all") or accepted & marked suspicious ("~all") the mail - more on that later. Please explain to me: How this is a problem with DirectBuy and not with your server? Yes, my server rejected the message. Why? Should it have rejected it or should it have let the message through? Answering that is going to be geeker than most want to read ... so proceed only if you have your propeller hat on and spinning. Lets look at directbuy's SPF records:
I'll copy the two relevant records here: 1: "v=spf1
include:spf.protection.outlook.com include:aspmx.sailthru.com
include:cust-spf.exacttarget.com ~all" So line 1 says the only servers that can send mail that looks like it comes from directbuy will be at the SPF records looked up at spf.protection.outlook.com, aspmx.sailthru.com, and cust-spf.exacttarget.com. In other words, we won't tell you which servers directly but refer you to other lists of other servers to decide if things are good or not good. So we (or the server) has to check the SPF records starting with spf.protection.outlook.com and if that has includes go through those and if those have includes go a level deeper. In the computer science world we call this "recursion". Here are the spf records at spf.protection.outlook.com:
Now he is giving us some IP addresses we can check against ... and another list (spfa.protection.outlook.com) to look at as well. spfa.protection.outlook.com:
Oh look! spfa wants us to include spfb.protection.outlook.com as well! (get the idea? This recursive stuff can go on and on...) Rather than recurse down that rabbit hole, I'm going to skip a few steps and look for the 'sailthru.com' which is what the server that was trying to send the password reset email claimed to be - see the 2nd print screen's 2nd line from the top of the article. Here is the SPF record for aspmx.salilthru.com:
"v=spf1 ip4:64.34.47.128/27 ip4:64.34.57.192/26 ip4:65.39.215.0/24 ip4:192.64.236.0/24 ip4:192.64.237.0/24 ip4:173.228.155.0/24 ip4:192.64.238.0/24 ip4:204.153.120.0/23 ip4:162.208.119.181 -all" DirectBuy's password reset email is coming in from 65.39.215.183 - look at the "source" column on the first print screen - and if you look at the line above that is the 3rd IP address listed. So DirectBuy's email server is allowed to send in email and claim it is from the domain directbuy.com! But wait - we aren't done yet. SPF was the first record type I checked - DirectBuy also specified a SenderID record, that is the one that looked like #2 - repeating both lines here: 1: "v=spf1
include:spf.protection.outlook.com include:aspmx.sailthru.com
include:cust-spf.exacttarget.com ~all" Line 2 has us checking spf.protection.outlook.com, is missing both aspmx.sailthru.com and spf.exacttarget.com, and adds ucctradingcorp.com and cust-senderid.exacttarget.com. Also notice the suffix "-all" - this says to hard fail if not on the list vs. "~all" that says to mark it as suspicious and pass it into the server anyway. Now lets traverse the spf2.0/pra path! Here are both:
No SPF or SenderID records at ucctradingcorp.com, so nothing to see there, folks! If you are going to tell systems how to know you are authentic, you need to be sure all the systems that can claim they are from you know how to prove their authenticity. So why is it listed as a place to check for SPF? A great question to ask whatever tech person set this up incorrectly. cust-senderid.exacttarget.com has a nice list: cust-senderid.exacttarget.com. 300 IN TXT "spf2.0/pra ip4:64.132.92.0/24 ip4:64.132.88.0/23 ip4:66.231.80.0/20 ip4:68.232.192.0/20 ip4:199.122.120.0/21 ip 4:207.67.38.0/24 " "ip4:207.67.98.192/27 ip4:207.250.68.0/24 ip4:209.43.22.0/28 ip4:198.245.80.0/20 ip4:136.147.128.0/20 ip4:136.147.176.0/20 ip4:13.111.0.0/16 ip4:161.71.34.0/24 ip4:161.71.35.0/24 -all" Look carefully - do you see 65.39.251.183 covered? Anything even close? In case you aren't familiar with subnet masks and their notation, I'll just tell you the answer: "No" And the "-all" says to hard fail if someone is sending and they aren't on the list - so this message must be from some evil spoofing phishing spamming group that is up to no good, so you should just reject the message. Which is exactly what my server did - what DirectBuy's configuration told my server to do - rejected the message. Because they told it to. This is a very common problem - some tech folks configure things based on wizards and how-to documents sometimes putting together multiple sources without understanding how each one works and you end up with interoperability issues. Since some or a lot of the systems appear to work, they will blame anything that doesn't work on the end user, or their email provider, or their interent service provider, or their computer's antivirus software, or anything else that lets them close the trouble ticket and hang up the phone call. I wasn't on the call but I can here one of the tech people trying a password reset, seeing it come in, and saying to a poor helpless user "Well it worked for me - I just tried it - so the problem must be on your end." Diving Deeper Down the Rabbit Hole! I'm totally done with discussing what was wrong - now I'm curious just how Exchange 2010 processes the SPF & Sender ID stuff - and I don't have to setup anything new, I already have the network trace to look through. Why? Because this is how to learn things - get in, play around, see how things operate and try to understand how it is thinking. If you know how things work to a deep enough level, when something goes completely wonky you'll have an idea of what could have gone wrong to cause those symptoms. Tweaking my filter - I could show just the DNS queries with the filter 'dns' or the exact conversation between our mail servers by filtering on tcp stream #2 (tcp.stream eq 2) ...so combining them I'll get visibility into how Exchange processes both! For the record, my filter is now: dns || (tcp.stream eq 2) There will be some redaction in these screen shots to eliminate any license / proprietary information.
229-231 The connection is first established between the two mail servers - the one doing the password reset mail (ny1mta-138.sailthru.com) and mine. Just prior and after (redacted lines 218, 237, and 238) are my server checking the Trend Micro "Evil Server Blocklist" to see if the server is listed. Also early on my server is checking the barracuda block list, spamcop, and spamhaus - if the server is listed on ANY of these block lists my server will likely just hang up the phone kind of like when you hear a robocaller is calling your phone. This happens VERY early in the conversation.
The two servers are talking normally to each other so far - this is early in the conversation. 304: During the conversation, Exchange sends a query for any TXT records:
and in 305 we get back all the TXT records - I only expanded the interesting ones, but there are 5 txt records:
Exchange now goes through the SPF2.0/pra txt record - apparently this is more important to him than the spf1 record is - somthing I did not know until just this moment, which is why I do this kind of analysis:
343: Query for the list for spf.proection.outlook.com 351: Query for cust-senderid-exacttarget.com and then the magic answer - none of those lists referenced the IP address of the sending server 65.39.215.183, so in packet 594 Excange sends back the error message "550 5.7.1 Sender ID (PRA) Not Permitted" At this point Exchange must have decided that there is no point in looking at the spf1 record or list of servers there - it already knows based on SenderID that this is a fraudulent email so give the error and tell them to go away. After a short delay (~2 seconds):
1292: The sending server sees the error message and realizes there is
no point in continuing to try to send - he has been called out as a
fraud, so he sends the "QUIT" command. Conclusion There you go - way more than you ever wanted to know about mail and a small bit of how mail is currently sometimes authenticated as coming from a legitimate server that is authorized to send mail for a particular domain. If and only if it is configured correctly. An interesting quesiton which I can't answer because DirectBuy doesn't have SenderID configured correctly is if Exchange would pass SenderID and then check the spf1 record list or does it say that 'if SenderID passes then don't bother checking the SPF record". In other words, this onion might have more than one layer. I could test this if they fix the first error and break their spf1 record in the same way. Oh the things I would do with infinte time on my hands. If you found this helpful, please send me a brief email -- one line will more than do. If I see people need, want, and / or use this kind of information that will encourage me to keep creating this kind of content. Whereas if I never hear from anyone, then why bother?
If you are having problems and need someone to help diagnose them, I can
be reached at the Contact link on the top left menu bar. David Soussan Copyright (C) 2019 DAS Computer Consultants, LTD. All rights reserved. |
||||
 |
||||
