Hi,

We have several databases running in ICE 4.0.6, each database consists of two IB tables and a few MyISAM tables (for housekeeping data that needs insert/update operations).

We’ve been busily uploading data using LOAD FILE, and everything appeared to be going ok, but we’ve just suffered what appears to be a catastrophic data loss in three of the databases.

It would appear that twice when the server has crashed (once due to a power loss, the second due to a system hang that required a power cycle), we have lost the contents of the T[AB]0000x.ctb files. This occurred while data was being uploaded.

Of course the actual crash can only have occurred while at most one of the two IB tables was being updated (as it takes two separate LOAD FILE commands to add data), but we’ve lost the contents of all of the files for both tables in each affected database.

The corrupted database tables now have most of the TA0000x.ctb and TB0000x.ctb files with zero bytes, but sometimes the first one or two files are 46 bytes (this would appear to be consistent with saving the initial header information but not the column name).

Obviously with both sets of files lost, we are unable to recover the database contents. This is extremely annoying as it represents weeks of data uploading.

I think this represents a serious weakness in ICE, as to paraphrase the quote, to lose one set of column definitions is unfortunate, but to lose both looks like carelessness! 

It is also not clear to me exactly why it is impossible to recreate the table structure to recover the files. I’ve dug a bit into the file format, and apart from having to scan the data files work out the number of data packs, there doesn’t appear to be anything in these files that is unrecoverable.

Regards,
Deryck Brown

It’s almost funny and ironic. This is the second time I am responding to this post, an unfortunate swipe across the touch pad, sent my browser to a different screen, loosing the paragraphs that I just wrote.  Let’s try this again,

It is very unfortunate what has happened.  Do you not already have a backup procedure in place?  It is these procedures that protect us from the unforeseen.  As you already know, our compression is one of our best features and to achieve that compression, it takes a lot of work under the covers.  Many of our ICE production users have major procedures in place to protect their data.  All technologies have good points, bad points and weaknesses.  I share with you the frustrations you are experiencing.

I will ping the engineers and ask them if I can uncover some more documentation for you about extracting more from the raw data files and see what I can get to you.  Thank you for letting me know what has happened.

Hi Craig,

Thanks for the quick response.

Of course setting up a proper backup procedure was on my to-do list…

However, I wasn’t expecting a system restart to cause the database to commit suicide. My main worry at this stage was data loss due to a hardware (i.e., disk) failure.

We haved been trying ICE for a few months, and this data load was our first attempt to move it from testing to production use.

Regards,
Deryck Brown

Nor would I.  I have pinged the engineers to see if there’s something more I can give you about the raw files.  As soon as I hear back from them, I will follow up again.

Hi !

The recent version (not released yet as ICE) is addressing the data persistency problem. I hope it will help ... I cannot tell about releasing schedule though.

A few questions.

1. Does Infobright Services start up?
2. when you query the table, select COUNT(*), what are the results
3. can you post a “SHOW TABLE STATUS;” results and see what you get
4. How are you determining their was data loss? (by 0 records or by the 0 byte files)
5. Do you have any of the data load files that were previously loaded before the crash?