 |
||||
         |
    |
|||
|
 |
|||
|
AT&T Tech folks: You can skip the background if you wish. Background: We live in a house built in 1929 with plaster walls. Wi-fi has a hard time going between three walls, so I had to put a second access point on the net in order to cover the house. The window frames are all made of steel. I'm guessing there is metal screen put up which the plaster adheres to, but haven't broken into a wall to see for sure. So what we have is a house with rooms that are weak faraday cages, which means RF signals have a hard time getting through them. |
||||
Consequently, our AT&T wireless service stinks when we are in the house. Not that Verizon was any better - it wasn't. And it is neither company's fault. In fact, I point out to people who complain about their wireless the Verizon commercials where that guy is walking around saying "Can you hear me now? Good!" - pay careful attention and you will notice something - he is always outside! Never inside. As soon as I caught wind of AT&T's indoor 3G microcell, I jumped on it. In one sentence, think of it as a miniature cellular phone tower you put in your house and it uses your wired high speed internet connection to send your phone's voice / text / data over the internet instead of your phone having to struggle to talk and listen to the cell tower that is so many miles away. There are many other sites that describe the product in way more detail, but that is a 20,000 foot view of the microcell. Mine has been flakey since day #1 when I took it out of the box. When it works, it works very well. But it tends to just go offline every 3-5 days. Flakey means "My cell phone won't see the microcell, rebooting the phone doesn't fix it, the only thing that fixes it is power cycling the microcell." After one too many reboots, I got frustrated with it and did some on-line research. Found some posts on various forums that claimed I had to re-load the firmware. Early versions had issues talking to the iPhone 5, which took a while for engineering to figure out and new firmware fixed that. This device has no GUI whatsoever other than 5 LEDs that light, blink, or change color. There is no user interface to the device, which means the user is pretty blind - other than the LEDs. Further research says the device will load up new firmware if you completely remove it from your AT&T wireless account and re-add it. So I did that. And it took forever to register - as in "Over 24 hours", and at least two power cycles after I'd waited over 6 hours each time. During one of those waits, I called AT&T tech support to see what they could see, and at the time it hadn't phoned home in 4 or so days with a GPS position so they very kindly agreed to send me a replacement even though mine was out of warranty. At that moment, we both thought my 3G microcell was dead. But, minor miracle, somehow my blinking LED microcell magically registered itself the next day. It still flaked out and went offline after 4 days, but it wasn't completely dead. A newly refurbished microcell! So the new one arrives - unboxed, registered with AT&T online, put it where the old one was, power it up - and wait. It gets GPS lock, sees the local network, ... and then little 4 antenna bar icon on the unit just keeps blinking and blinking and blinking, about 2 seconds between blinks. I did some measurements and created this animated GIF - the new non-registering microcell is on the left, my original microcell is on the right and not powered, and my lovely 10 Mb/s hub I use for sniffing data on the wire is sitting under them.
Note to self: Window blinds make a wonder analog brightness adjustment, allowing perfect balance between the LEDs and the ambient lighting necessary to see everything correctly! Normally the unit wouldn't be upstairs, nor be connected via a network hub but a gigabit Ethernet switch, nor sit on a radiator. But I wanted a decent portrait showing the LEDs in case there is some hidden intelligence in there AT&T or someone else finding this page finds useful. So the unit does everything except find the AT&T wireless network to phone into - at least that is what I thought seeing antenna bars blinking. And in fact that is what AT&T tech support tried to tell me. Another tech said that meant the microcell cannot see my network, that I have a network problem. But they are wrong. They are 100% wrong. One meaning for this flashing light pattern is the microcell is trying to phone home over the internet and the microcell is not seeing any response from the system on the other side of the internet the microcell is trying to talk to. There are many reasons this can happen, including all those things tech support wanted me to try. But the problem can also be because of an issue with AT&T, and the flat out refusal to even consider that as a possibility until I've completely disrupted my world is completely inexcusable. In fact, if the servers the microcell is trying to talk to are powered off, the lights will blink with the exact same pattern as if the microcell can't get out of my location and onto the internet. The lights alone can't tell the difference (I could make them tell the difference easily! Just give me the source code and a couple of days... seriously...). Tech support can't tell the difference, but I can - because I looked at the data coming out of the microcell. Data on the wire doesn't lie. Background is over, now for the juicy bits! My first thought seeing blinking antennas was to move the unit to another part of the house - maybe a tower was down and it couldn't shoot RF signals in the direction I had it facing, so it moved upstairs and to the other side of the house. No difference. I thought as a microcell it might actually talk to another cell tower and not just my cell phone. Still not sure if that is true or not - one of the tech support people on the phone said they saw one of the indicators go green when I did yet another power cycle, and the only way they could have seen that data was over the air and not over the internet. So last night fed up with a new unit arriving and not working, I started digging in. If you've read any of my other diagnostic articles, you know my first tool of choice is often a network sniff. If I can see the packets and who is talking to what, I can pretty quickly isolate a problem and know what direction to go looking for a solution at. And this device is just that - a networked bit of equipment that has to talk to other things to function properly. So I took both units, hooked them into a network hub, hooked the laptop up with Wireshark, and watched the data as both units powered up, acquired an IP address, and attempted to talk to their home bases over the Internet. Knowing what I learned below, I contacted AT&T technical support, spoke to the 3G microcell folks. They had me do all kinds of stuff that were completely useless in fixing the problem - some of these activities included: Power cycling (Been There, Done That, didn't fix it any of the last
8+ times...) One of the agents in the support chain whose name will not be mentioned as I don't want to embarrass her insisted that all the lights on the front of the microcell had to be on including the computer light, and the fact it wasn't lit said I had a network or connection problem I would have to fix before she could continue diagnosing the activation problem. Not true! AT&T Tech folks: You should really read your own support page, this is from https://www.att.com/devicehowto/?make=ATT&model=3GMicroCell :
In all fairness, I understand and sympathize with tech support folks. They probably have to deal with phone calls like "My brother in California can't see my 3G Microcell here in Michigan, why isn't it working?" all day, tirelessly and cheerfully explaining "Sir, the 3G microcell only has a range of around 1000 feet and can't make it the 2000 or so miles across the United States. Thank you for choosing AT&T!" I would gladly pay $2500 every year to be spread out among technical support companies for this:
(if you like that, you need to spend a few hours at http://xkcd.com and enjoy. This comic is at http://xkcd.com/806/) Ok, so now I'm going to get the propeller on top of my hat spinning full speed. Hang on! A sniff of what the new brain-dead replacement microcell is doing This is the just received AT&T replacement 3G microcell trying to talk back to AT&T via the Internet. Network sniff was taken 4/1/2013 near midnight. All screen shots are thumbnails and can be made larger by clicking on them. First I'll show you the replacement failing microcell, then my original microcell that still works. I have eliminated most of the unimportant packets from this screen capture.
Interpretation for those not quite as intimate with network packets as I am: #2505 & 2512: The microcell is getting the time via Network Time Protocol from the device at 12.230.208.48. It can talk fine and hear back the time fine - this means the device can talk to the internet just fine. This means no amount of resetting or power cycling my local router, firewall, or any other equipment will fix the problem. #2783: The microcell wants to phone home to AT&T, so it does a DNS query to find where home is and looks for the address at dpese.wireless.att.com #2785: DNS answers that dpese.wireless.att.com
resolves to 12.230.208.67 or 12.230.208.40 Independent proof that 12.230.208.40 and 12.230.208.67 are both completely offline as far as the 3G microcell is concerned:
(The microcell tries to establish a TCP socket on port 443 back to the server. My Telnet command is trying to do the same thing. If the server was alive, reachable, and answering the screen would have cleared to a black screen as telnet sees the connection is established. Not that I can continue with the TLS handshake, but I can at least see if the server is responding at all or not.) Here is the consumer status screen showing that it hasn't phoned home. This is from this morning, 4/2/2013, no change from anytime yesterday.
Since the microcell is trying to talk to an AT&T server that does not currently answer, the AT&T network is blind as to the status of this microcell. The only way AT&T can talk to the device is over the radio airwaves, at least until AT&T fixes the server this microcell is trying to talk to. The microcell is calling over the internet to a pair of servers that are not responding. That is the problem - not my internet connection, not my router, not my firewall, not my power supply, not the air in the house, not my Ethernet cable, not the lights on the 3G microcell, not the position in the window, the phase of the moon, or the sun shining on the microcell .... Double-proof: This new microcell is trying to talk to
dpese.wireless.att.com resolves to 12.230.208.67 or
12.230.208.40. If the microcell can't talk to the first but can talk to the second, those two IP addresses are close enough they are likely in the same datacenter. Does it make sense if my microcell cannot talk to the first device but can talk to the second device that the problem is anywhere in my network? Think about that. A sniff of what the old microcell is doing - this one will register. So here is the old microcell, that will register just fine. It is using the same ethernet cable, same power supply, same internet feed, same position in the window, same wooden window frame it is physically sitting on, and breathing the same air as the other microcell. I'll walk you through exactly how it is supposed to work...
A similar pattern to what is seen in the first sniff, except this one works. So why do I want a new one? Because it flakes out and goes offline every 3-4 days, requiring a hard power cycle and occasionally requiring removal and re-registration. It is flakey. Has been since day #1. Yes, I said that in the background already - but I wrote this part before deciding this made a great "How to debug complex things nobody else can figure out" article and needed background material above. Anyway, here is what this box is doing. #589-590: microcell is talking to 12.230.209.5 to do the NTP time synchronization stuff. #595: This box is clearly different - probably running different firmware. It wants to talk to dpece-hazelwood.wireless.att.com - note this is not the same DNS name as the new microcell wants to talk to! #596: DNS answers back that dpece-hazelwood.wireless.att.com is at a 12.230.209.72 and 12.230.209.71 - note these are different addresses than the replacement microcell resolves to. #604-623, and onward: This is the proper TCP handshake - SYN packet from the microcell, SYN-ACK from the server at 12.230.209.71, and an ACK from the microcell back to 12.230.209.71. This is working because 12.230.209.71 is alive and talking. Conclusion: This is an AT&T server or AT&T DNS record issue. Either BOTH the servers at dpese.wireless.att.com which resolve to 12.230.208.67 and 12.230.208.40 are not online and running the services that respond to microcell communications via SSL on port 443, or the addresses pointed to by the AT&T DNS Servers that resolve the addresses of the name dpese.wireless.att.com are pointing to the wrong addresses. Fix it already. Stop making me jump through your diagnostic hoops that will accomplish absolutely nothing. Epilogue: Days later, I got a call from someone asking my status - I asked "Have you read the posted data?" - which they then did. He did some research, said I was in 3 different support paths, and there wasn't much more he could do that those support folks were already on it. So when he asked if there is anything else he could do, I asked "Do you have the ability to setup to swap out this unit for yet another one? Hopefully one that is configured to talk to a server that is alive instead of one that is dead?" And short story, a couple of days later I had another microcell ... and this one registers just fine! And so far this unit has gone months without losing connectivity ... I'm calling this problem solved! Final Words... If you found this helpful or not, please send me a brief email -- one
line will more than do. If I saved you a bunch of time (and thus $$),
and you wanted to show appreciation, sending a little love via PayPal or
an Amazon gift card is also very much appreciated!
|
||||
 |
||||
