We have several databases running in ICE 4.0.6, each database consists of two IB tables and a few MyISAM tables (for housekeeping data that needs insert/update operations).
We’ve been busily uploading data using LOAD FILE, and everything appeared to be going ok, but we’ve just suffered what appears to be a catastrophic data loss in three of the databases.
It would appear that twice when the server has crashed (once due to a power loss, the second due to a system hang that required a power cycle), we have lost the contents of the T[AB]0000x.ctb files. This occurred while data was being uploaded.
Of course the actual crash can only have occurred while at most one of the two IB tables was being updated (as it takes two separate LOAD FILE commands to add data), but we’ve lost the contents of all of the files for both tables in each affected database.
The corrupted database tables now have most of the TA0000x.ctb and TB0000x.ctb files with zero bytes, but sometimes the first one or two files are 46 bytes (this would appear to be consistent with saving the initial header information but not the column name).
Obviously with both sets of files lost, we are unable to recover the database contents. This is extremely annoying as it represents weeks of data uploading.
I think this represents a serious weakness in ICE, as to paraphrase the quote, to lose one set of column definitions is unfortunate, but to lose both looks like carelessness!
It is also not clear to me exactly why it is impossible to recreate the table structure to recover the files. I’ve dug a bit into the file format, and apart from having to scan the data files work out the number of data packs, there doesn’t appear to be anything in these files that is unrecoverable.