It all started with a "phone broker". They put together a package with an Internet Service provider and a VOIP provider. The problem was that neither one has any idea what the other's job entails.
The ISP brings a T1 into the premise, installs the CPE/router, and hands off responsibility to the customer. So far, not bad.
The VOIP provider brings in a VOIP/router/firewall appliance, connects it up with a default configuration, tests the phones, tests for Internet connectivity and leaves. It's after this that the experience gets a bit scary.
The problem is that many of your VOIP installers are the BOC's traditional voice jockeys, or closely related descendants. These folks are terrific when it comes to analog signals but most don't know a bit from a byte, much less know anything about TCP/IP, security practices, nor how their customers' data processing needs fit with the equipment they have just installed.
Soon enough, customer wants to make some IP services available to the outside world and needs some routing and firewall changes made. So I call the ISP and mention that I need some NAT and/or PAT routes, and ISP says "what NAT?" I know the customer's internal IP space uses NAT so I fish around and find out they have a VOIP appliance behind the CPE doing the NAT. Progress! So I call the VOIP installer and ask him to create some NAT and/or PAT routes through the appliance to an internal IP and I get silence. Then some stuttering. I can literally see the resulting puzzlement through the telephone. Oh, nuts, I might as well be speaking Martian.
Frustrated, I ask of Mr. VOIP the exact brand/model of appliance he installed, then hang up the phone and start fishing around through Google. No more than five minutes later I have a document that mentions the factory default password for said appliance. From the comfort of home I fire up a web browser, punch in the customer's IP, and up comes the appliance administration web page. Key in the factory default account name and password and IT WORKS.
OH MY DEAR GOD IN HEAVEN, these buffoons left the appliance open to the entire Internet with its FACTORY DEFAULT PASSWORD in place. And it was NO ACCIDENT.
I make the necessary routing and firewall changes, confirm that it all works and that no damage was done, then change the password to something secure and inform the customer of the new password.
A week or two later I get an angry call from Mr. VOIP asking why I changed the admin password on the appliance. After I was stupid enough to actually try to respond to such an idiotic question, he insisted that there was no need for me to change the password because - now get this - "the only way into the appliance is through a serial port connection on the device itself or via secure shell from inside the customer's LAN". And he further vehemently insisted that the password MUST be put back to the default because they "CANNOT support it unless the password is set to the default".
Laughable stuff this would be, if it weren't so terribly, pathetically ignorant.
Roll the tape forward a few weeks. Customer having some voice issues, asks me if there were any more changes and is there anything they need to know before they call for help. I haven't touched their stuff since day two, but it should only take me a few seconds at a web browser to check the status of the appliance, right? I open the admin web page, try the password I set a few weeks before, and it doesn't work. Type again more carefully, still no good. Try the factory default password just for laughs, and that doesn't work either.
Perplexed, I ask the customer about the appliance password. "Oh, they (the VOIP installer) changed it and won't tell us what the new password is".