Verizon FiOS outbound mail TLS problem - Superpages people here?

Sat Jun 6 23:20:20 UTC 2015

Oh, and the way we narrowed it down was somewhat oblique. Because their logs said a TLS connection was established we had a hard time convincing them it wasn't. They were convinced it was us who was broke.

We had to send them a PCAP and then they ran one and got the same results. We were communicating via their IronPort "secure email" system and I noticed that the Cisco copyright notice on their messages was from 2012. That put me on the path to look at the Cisco release notes. Once I pointed out that they seemed to be a bit behind and there were fixes in later versions, the conversation went in a different direction. :-)
> From: sixsigma44 at hotmail.com
> To: blake at ispn.net; nanog at nanog.org
> Subject: RE: Verizon FiOS outbound mail TLS problem - Superpages people here?
> Date: Sat, 6 Jun 2015 19:13:38 -0400
> 
> We had a similar issue around November last year where an upgrade on our
>  PostFix MTA to a current version of OpenSSL, which has Mandatory TLS 
> enabled for certain recipient domains, suddenly started generating the 
> same errors with just one recipient domain.
> 
> We eventually figured
>  out that the problem was they were running an outdated version of the 
> AsyncOS on their Cisco IronPorts. Firmware versions prior to 8.02 had 
> several problems with TLS and one of them was an inability to 
> interoperate with senders who used a newer version of OpenSSL. Their 
> IronPort logs in fact showed a TLS connection was established when it 
> wasn't. (We had switched them to Opportunistic TLS to be able to send 
> emails but their logs still showed TLS while a PCAP showed clear text 
> SMTP.)
> 
> As soon as that company updated their IronPorts to a v8.5 
> variant the problem went away. They would not tell us what version they 
> used to run but did confirm it was prior to v8.02.
> 
> Interestingly, www.checktls.com
>  said they were OK. The admins at Check TLS confirmed that, at that time
>  (the end of 2014), they were running a version of OpenSSL on their 
> website that was still compatible with the older AsyncOS version. 
> 
> FWIW,
> 
> Ray
> > Date: Thu, 4 Jun 2015 11:46:35 -0500
> > From: blake at ispn.net
> > To: nanog at nanog.org
> > Subject: Re: Verizon FiOS outbound mail TLS problem - Superpages people here?
> > 
> > I have no relation, but as a mail server operator I can say that I 
> > wouldn't be surprised if this is actually a TLS version mismatch or 
> > intolerance problem. I would suggest ensuring that both ends support TLS 
> > 1.0, 1.1, and 1.2 and use version tolerant TLS implementations. Next on 
> > the short list would be not having compatible cyphers between the two 
> > servers.
> > 
> > Either way, since the error was a 403 error, the expected behavior would 
> > be to queue and retry in plain text; Sounds like a broken MTA 
> > implementation or misconfiguration if the sending servers do not revert 
> > to plain text.
> > 
> > --Blake
> > 
> > Jay Ashworth wrote on 6/4/2015 11:15 AM:
> > > Anyone on the list who does outbound delivery for Verizon (which I think
> > > is actually Superpages)?  A client has smart-hosted outbounds to *one*
> > > of his customers bouncing suddenly with
> > >
> > >    Deferred: 403 4.7.0 TLS handshake failed.
> > >
> > > *My* inclination is to think that a cert expired somewhere, but his non-tech
> > > contact there tells him that the tech people think things are ok.
> > >
> > > I'm trying to get a mailer log fragment from them.
> > >
> > > Cheers,
> > > -- jra
> > >
> > 
>  		 	   		  