First of all I just want to say “thank you” to everybody who posted his or her questions and answers concerning Talend connector. Forum was THE place to find important clues and bits to get the Talend run with Infobright.

Newer the less, the system is running but the loading speed I do observe is still far away from the reported numbers in other configurations. That said, the Engine load 90 lines a 100 cols in about 3 sec if loaded with “load data infile ...” sql command. If submitted through Talend connector the system will never load more than 700 rows per second. This means almost two orders of magnitude difference in loading speed.

According to the docs the Talend connector is using named pipe feeding the bulk loader which shall work incrediby fast. But either java, or my computer or some other wrong option in the database is throttling the load speed.

Does anybody else share the same expirience?

I am using ICE 3.3.1 and Talend Open Studio 3.2

Sincerely
Nikola

P.S. Please appologise my not realy good english

Hi,

We have an action item to look at the remote load performance.

Have you tried loading locally (with Infobright and Talend on localhost)? It should be much faster.

Geoffrey

Hi,

thanks for the fast response.

Both Talend instance and ICE run on the same maschine. Albeit, it is a simple desktop wit 2G ram and sata drive.

I have already tested different Brigthouse.ini options giving more ram access to the ICE engine and loader. Unfortunately that seam not to be the reason. Giving more memmory does not change the performace rapidly.

On the other side, the throutput of talend seam not to be the bottelneck. If written to a CSV file, from the same datasource talend job reaches around 20000 row/s.

Could it be that named_pipes system in the mysql server or in the operating system can be configured / influenced by any options??

Sincerely
Nikola

Hi !

With only 2GB RAM it is likely that your system is swapping heavily. Use “top” under Linux or “Task Manager” in Windows to see memory usage and swapping activity.

Both Talend instance and ICE run on the same machine

Make sure you have “localhost” or “127.0.0.1” in the hostname field in the connector (not the machine’s external hostname or DNS name). This will ensure that it can use the local named pipe.

Geoffrey