the first Chilean miner just emerged - me, too

Pretty amazing.

I just emerged from two days of feeling somewhat underground.

It happened yesterday at 12:46 pm. I was running what I thought was the first test of updating the email address on one Salesforce contact record. Imagine my horror when, ten seconds after sending the SQL thru the internet to the Salesforce server, I realized that I had just submitted the command to update EVERY email address, not just the one person's I had intended.

I trembled as I killed the Access process that had issued the command, praying that Salesforce would see that its originator was gone and STOP before too much damage was done, not knowing if it would stop before it had clobbered 35,000 records.

After I convinced myself I had done everything in my power, I logged back in and ran a test. 4100 Contact records now had the same email address, of my test person. Could be worse, but very bad.

I called the client and left voice messages, explaining that I had caused an emergency, and they needed to get everybody out of Salesforce until the backup could be restored. At the worst, I thought, any work done that (Monday) morning would have to be redone.

After some minutes of calling, I connected with a live person. They started making inquiries about a restore.

After a half-hour, it was revealed that the consultant who set up Salesforce for them had neglected to turn on the auto-backup function, and had never told them about its existence.

I drove over to the client, talked about options, grabbed the final version of the database which had supplied Salesforce with its converted data, and came home.

After some exploratory digging, I sadly concluded that writing an automated mass update was problematic, due to the many combinations of old and new data. There was nothing for it but to manually fix as much as possible.

I sat at my computer for the next 10 hours, manually examining each person's data and applying updates. It was horrifyingly slow. By midnight, I had resolved about a thousand cases.

I resumed this morning around 7. By 3 this afternoon, I was down to under 1700 left, and had concluded that it would take 2 more days to resolve the rest, and that the bulk of cases were where the bad email address had overlaid no address, rather than an existing good address.

I built a mass-update pass-thru query, to erase all remaining instances of the bad email address - it ran in about 45 seconds and I was done.

I walked outside, feeling like a miner who had been underground for two days, not to mention two months.

Now I can resume the program I was working on at the time of the catastrophe - which, as folks order trees in the coming months, will correctly update their good email addresses into Salesforce. Over time, the results of my one moment of programming inattention will diminish.

This has been the worst data-bungle of my 38-year career. Prior to that, the worst bungle occurred way back in my 2nd year as a computer programmer, which I may write about someday, as if you cared.

Congratulations to the engineers in Chile, who figured out what needed to be done, and, apparently, are about to deliver a bunch of men from what must have seemed to be, when it occurred, a horrible, silent, death sentence.

Un milagro!