[AC-Admins] Lion outage

[Simba]

On 11/25/2015 01:05 PM, Pippin Bear wrote:
> Anyone know if lion's admins know that it's down?  I mean... I'm
> sure they do, but:

      I was painfully aware that it was down (along with all of my
company's hosting customers [and their customers {and their customers'
customers}]).

      Someone managed to fraudulently (but successfully) submit
a disconnect / cancellation order to our uplink (CenturyLink) for our
circuit and the associated account on the 12th.  CenturyLink's internal
disconnect order went in on Friday, but the main contacts for the
account weren't notified (or there was just outdated contact
information because that circuit is 10 years old). Nonetheless,
CenturyLink fully decommissioned and removed the provisioning on the
circuit before we had any indication that anything was happening.  This
meant that to get it to be reconnected, they needed to fully provision
the circuit as if it was a brand new order.  Given that, and the fact
that this is a holiday week, so they were short staffed, this led to a
45 hour down time.

-- 
Simba