1) after few months working in a bank, I was doing some simple admin check task via RDP to a Windows 2003 (no, maybe 2000) server, when I right-clicked the network icon and instead of clicking the properties options i clicked "disable". Just the time to say "oh sh!t" and to realise that it was the production Trading On Line machine, on a remote datacenter, during market hours, and to discover couple of minutes later that the KVM over IP was crappy and was not working. We had to call the datacenter operators to go back to the local KVM and re-enable the NIC.
Lesson 1: Better move slowly when you're on a production machine (and also have plan B and C to reach your machines is a good idea)
2) same bank, one or two years later, I was doing some testing on a new mail system that integrated also VoIP (SIP). Mail/SIP System running in a VM (I think Vmware Server at that time) in the same remote datacenter as above. So, I enable the SIP feature and after few seconds, bum, we lose the whole (production) datacenter and the connection between the local server room and the datacenter.
Panic, I look at my colleague, WTF in stereo, everything come back for few sec, bum again down.
Long story short, the issue was that that version of Netscreen firewall ScreenOS had a buggy ALG implementation for SIP that lead to core dumps.
The fun thing is that we had two of those in HA, same version of course, so they were bouncing between core dumping, rebooting slave becoming master and then core dumping again etc..
We had to ask a datacenter operator to reach the rack, disconnect one of the cables from the firewall (the one that was managing the traffic of the DMZ where that machine was hosted) and then reach the virtual host to kill the machine.
Lesson 2: you can segment your network but if everything is connected through the same device(s), sh!t can still hit the fan...
1) after few months working in a bank, I was doing some simple admin check task via RDP to a Windows 2003 (no, maybe 2000) server, when I right-clicked the network icon and instead of clicking the properties options i clicked "disable". Just the time to say "oh sh!t" and to realise that it was the production Trading On Line machine, on a remote datacenter, during market hours, and to discover couple of minutes later that the KVM over IP was crappy and was not working. We had to call the datacenter operators to go back to the local KVM and re-enable the NIC.
Lesson 1: Better move slowly when you're on a production machine (and also have plan B and C to reach your machines is a good idea)
2) same bank, one or two years later, I was doing some testing on a new mail system that integrated also VoIP (SIP). Mail/SIP System running in a VM (I think Vmware Server at that time) in the same remote datacenter as above. So, I enable the SIP feature and after few seconds, bum, we lose the whole (production) datacenter and the connection between the local server room and the datacenter. Panic, I look at my colleague, WTF in stereo, everything come back for few sec, bum again down. Long story short, the issue was that that version of Netscreen firewall ScreenOS had a buggy ALG implementation for SIP that lead to core dumps. The fun thing is that we had two of those in HA, same version of course, so they were bouncing between core dumping, rebooting slave becoming master and then core dumping again etc.. We had to ask a datacenter operator to reach the rack, disconnect one of the cables from the firewall (the one that was managing the traffic of the DMZ where that machine was hosted) and then reach the virtual host to kill the machine.
Lesson 2: you can segment your network but if everything is connected through the same device(s), sh!t can still hit the fan...