I am attempting to load some data into ICE to investigate the feasibility of using ICE or IEE on a project.  So far I’ve loaded 2,820,301 rows.  I was attempting to load some more data, a file with 1,695,093 records, when I received the error “The Infobright storage engine has encountered an unexpected error”.  Looking at the log file shows:

2010-08-16 04:44:11 The Infobright storage engine has encountered an unexpected error. The current transaction has been rolled back.
2010-08-16 04:44:11 Error: Wrong data or column definition. Row: 375761, field: 56.

Looking at row 375761 of the input file shows nothing out of the ordinary.  The raw is nearly identical to the one preceding it.  The field in question is NULL, like many before it, so it should cause no problem.

I can copy row 375761 into its own file and load it just fine, so its not an error in the data or column definition.

Any suggestions on how to get a better idea of what the real problem is?

Reading another post on the forums that suggested breaking up the file, I split the file into chunks with no more than 300,000 lines.  I was successful loading each chunk.  Thus, the problem was not in the data. 

Infobright looks very promising, but I can’t see this type of obscure errors make me feel very secure using it in a production environment.

I can seriously say or dare to say that the problem is the data that you have in the input file, It could be because of row or field delimeters in the fileld value. Note: The way you copy data from a text editor and the way you insert into a database differs(search for this).. Forgive me, if I am wrong. Infobright should focus on more options on import and export of data as in mysql, otherwise it keeps hurting people like me when we hear that Infobright looks very promising, when it actually is promising. People don’t intentionally mean to say this, but they fall(lack of experience in what they are doing/misunderstanding of what the system actually supports/Once in a blue moon atleast 1% Bugs exist in every system) in such a situation to say this.

Hi Buddoda,

Thanks for your response.  What would you recommend for improvements to the loader?  You mention that its experiences like this which cause people to resort back to their previous behavior, so I hope to nix this type of issue in the future.

Feel free to reply or skype me (jeff.at.infobright).

Cheers,

-/j