hi all,

i am using INFOBRIFHT 3.2(enterprise trail version) as my database and Talend 3.1.1 as my ETL tool.

i have created a job in talend. it will process 50 mill records from source and target will be 45 mill. 


in my job inserts in infobright are very slow.(837 rows/sec)
if i am writing same output in file - its speed is - 28517 rows/sec.

mysql(innodb)  insert speed is 11000 rows/sec .

can you give me an idea to speed up my script.

Hi Raja,

Have you tried installing the Talend/Infobright connector? It connects directly to our fast bulk loader. You will probably see better load speed than with MySQL Output.

You can get it from the Infobright contributed software page at http://www.infobright.org/Downloads/Contributed-Software/

Geoffrey

Thank you Geoffrey for our quick reply.

thanks,
RAJA K

hi Geoffrey,

My problem was solved.Thank You very much for your help.

Now i processed 23 mill from source and 20 mill are inserted in Target Infobright DB.
it took only 20 min for executing my total script.

Inserts in infobright with this connector is faster than the tMysqloutput for my script.

Thank you for valuable suggestion.

thanks,
Raja K

my method
mysql> select * from t into outfile ‘d:/infobright/1M.csv’;
Query OK, 1000000 rows affected, 1 warning (2.12 sec)

mysql> load data infile ‘f:/soft/hu41.csv’ into table hu41 fields terminated by ‘,’ ;
Query OK, 24590997 rows affected (7 min 58.69 sec)

Thanks, Tom.  Glad it worked.  Let us know if you run into any further questions.

Cheers,

Jeff