Joinutility seperatorLogin utility separator Infobright.com
   
 
Load Data Performance
Posted: 20 May 2009 08:49 AM   Ignore ]  
Newbie
Rank
Total Posts:  46
Joined  2009-05-19

Hi

I am doing some testing with ICE (vs partitioned table in MySQL) and have just done an upload.

File size:      5.3GB
Row count: 34329032

Time taken : 1030 seconds

CREATE TABLE `from_table` (
  `
uidvarbinary(20NOT NULL,
  `
datedatetime NOT NULL,
  `
emailchar(255NOT NULL,
  `
useridchar(255NOT NULL,
  `
subjectchar(255NOT NULL,
  `
bytesint(11NOT NULL
ENGINE=BRIGHTHOUSE DEFAULT CHARSET=ascii 

Running on 64-bit Fedora 9 with two dual-cores (XEON), 12GB RAM

How does that look to the experts? I notice it wasn’t doing any threading across multiple cores. Should I expect it to?

Anything I should do to tune/check the hardware/disks? Or does these figures look ok already?

The MySQL test is still running grin I will update when it finishes, hah. There are indexes on the table and that is what I think is killing the time

Thanks

Profile
 
Posted: 20 May 2009 10:07 AM   Ignore ]   [ # 1 ]  
Sr. Member
Avatar
RankRankRankRank
Total Posts:  651
Joined  2008-08-18

Hi !

ICE loader does not use multithreading, IEE (commercial) does.

Some numbers (load speed) are mentioned in e.g.  http://www.infobright.org/Forums/viewthread/375/ and http://www.infobright.org/Forums/viewthread/425/

Load is usually faster for loading numerics. Loading long strings benefits much from multithreading (much faster in IEE).

Disk tuning can be necessary if loading a few tables in parallel.

Profile