Joinutility seperatorLogin utility separator Infobright.com
   
 
LOAD DATA INFILE string truncation issues
Posted: 25 February 2010 09:12 AM   Ignore ]  
Jr. Member
Avatar
RankRank
Total Posts:  58
Joined  2009-10-07

Hi,

this is what I came across today. I’m trying to load file with some strings to a table (ENGINE=BRIGHTHOUSE) with a varchar(255) field. File has strings up to, say, 300 characters long.
When using DEFAULT CHARSET=utf8, strings load just fine without any warnings, i.e. without truncation. So at the end I have a column designed to hold up to 255 characters containing a lot more data. Well, I guess I’m not sure if utf8 support is properly implemented yet.
When using DEFAULT CHARSET=latin1, I get this:
ERROR 2 (HY000): Wrong data or column definition. Row: 1, field: 1.

Which is also painful because it breaks automated data load (or requires me to validate my data manually).
Could you please comment on
- whether utf8 is officially supported or not, and if not, are there are plans to support it in the nearest future
- is there a way to enable data truncation instead of throwing an error

Thanks.

Profile
 
Posted: 25 February 2010 01:43 PM   Ignore ]   [ # 1 ]  
Newbie
Rank
Total Posts:  15
Joined  2009-06-19

UTF-8 support depends on which version you are using. IEE 3.3.1 supports UTF-8, but ICE 3.3.1 does not support it yet, as far as I know it should be supported in the next version.
To answer you question about truncation, it do not think there are anyway to this this at the moment, but I could be wrong.

Signature 
Profile
 
Posted: 25 February 2010 02:41 PM   Ignore ]   [ # 2 ]  
Sr. Member
Avatar
RankRankRankRank
Total Posts:  738
Joined  2008-08-18

Two alternative TRUNCATE options have been posted here. One is a bash script, the other is a stored procedure.  Note: To use the stored procedure, it must be created in the same schema where it is used.

Signature 
Profile
 
Posted: 26 February 2010 06:24 AM   Ignore ]   [ # 3 ]  
Jr. Member
Avatar
RankRank
Total Posts:  58
Joined  2009-10-07

Thanks for your comments everyone. I’ll look into the suggested truncation options. However would be nice to have an automated way for that one day, alike MyISAM/InnoDB. Anyway, thanks again!

Profile