Over the last few days I program I was working on has been showing very inconsistent and almost nonsense errors.
Of UTF8 Errors
At first some XML files were being reported as not propertly UTF-8 encoded. A fault in the program we tried to fix. The program does some automatic conversion, so I revise that code and made some changes. But thinks kept getting worse. Also, restarting the server twice, and reading the same files (no changes in between) lead to different errors.
Of Strange Uppercase Letters
As we kept looking in the logs, the XML processing errors increased and start to invocle also XML tags. An open tag was not properly closed. Or tags would not match. Or:
Parse Error: Opening and ending tag mismatch: tweetId line 0 and twEetId
The closign tag has an uppercase E. Now these files are not written by a human, they are generated by the program. Many 'e' characters got turned uppercase, causing trouble. Than some 'm' characters. Since there was some code to code with ASCII characters in UTF8 files, I though that code had gone bonkers. So I replaced it with some totally different code. Same effect.
Strings in a String List That Change By Themselves
Finally, today we had a file name error. The program loads a list of files in a TStringList, than reads some of the files. One of the files was missing, well no, the original file was there but the file names in the TStringList got a letter turned to uppercase. Since the storage is on a Linux box, file names are cese sensitive. This was completely unbelievable, but it made an earleir suspicious (hardware problems?) seems plausible.
We run a MemTest. We got droves of errors. Asked the web farm support people to fix the hardware... and within a couple of hours we had the RAM replaced and the application running nicely with no error (well, only a few minor errors that are in the code!).
It's "a Bit in the RAM", Can You Believe It?
I was a little shocked. I had witnesses disk drive faillures. I had witmesses RAM errors (writing in an area would cause access violations and crash programs). But I would have never thought that a hardware errror in one of the bits of some bytes of a memroy block will turn some letters uppercase! That's an hardware based algorithm, very fast.... but unfortunately not very reliable.
Were someone to tell me letters in his program were magically turning uppercase, I'd have probably told him to start a career in something other than programming. But since I witnessed this, I realize there is no limit in the creativity of hardware failures!
PS. No, no error on this server, I wrote uppercase letters in the title on purpose, as you probably guessed!