Infobright.org Forums

Load Data Into Infobright Community Edition using Third Party ETL Tool Like Talend

2012-02-03T04:44:51Z

Hi All,

Can any one tell me How to load data Into Infobright Community Edition db using third party ETL tool like Talend ?

Regards,
Rajeev Kumar Mehta

Best ETL tool for Infobright

2010-07-05T20:01:15Z

Hi All,

Can you suggest me the BEST ETL tool which can be used to load data from SQL Server/DB2/Oracle (in scheduled manner) into Infobright?

Use Infobright Loader and ignore error

2012-01-10T11:51:57Z

Hi all,

I’m trying to load a file using infobright loader

LOAD DATA INFILE '/tmp/filetodump' INTO TABLE data_20120110 FIELDS TERMINATED BY ',' ENCLOSED BY ''; 

I got the following error

ERROR 2 (HY000): Wrong data or column definition. Row: 82, field: 36. 

If i check my file, there is indeed an error at row 82. However, i’d like to continue to load other part of my file. So my question is, do you think it’s possible to load a file, ignoring row in errors ?
I know it’s possible with oracle.

No Network in VM ICE + Talend + JasperSoft

2012-02-03T06:01:38Z

Hi All,

Can any one tell me how to browse internet on VM ICE + Talend + JasperSoft.
Any configuration needed ?
I have install this VM on windows XP machine with internet connection but i am unable to browse internet in the VM.

Thanks,
Rajeev Kumar Mehta

ELT vs ETL

2012-01-02T08:34:02Z

Hello,

Are there any experiences or best practices regarding the transformation of large data sets (~100Mio records)?

With an Oracle DB, I made good experiences with Load and then Transform instead of doing an external Transform and then the Load. The “trick” was using cascaded “create table as select” statements to join and filter a “base input table” in consecutive steps with data from several other, large tables (denormalizing and filtering).

So in effect I copied data from table to table within the database.

Is that also a good idea with Infobright (or columnar storage in general)?
Or is it faster to preprocess the data externally (e.g. using Pentaho/Kettle “table input” and “merge join” transformation steps)?

Best regards,

Bernd

Example Using Core Connector Lib

2011-12-28T09:23:50Z

Hello,

I need to load delimited files that have:

* two character column delimiters
* three character record delimiters (none of them CR or LF)
* contain potentially CR/LF characters in the data fields

The given LOAD command can’t handle this.
Pentaho/Kettle can’t handle it either (CR or LF is a “must have” as record delimiter).

Currently I need to run under Windows.

So I plan to write a custom loader/filter for these (large) files (In Java or C++).
I don’t want to create intermediate files and thought it might be a good idea to use the core connector lib.
I had a short look into the source but couldn’t find a starting point.

Is there an example on how to use/call the lib?

I think InfobrightNamedPipeLoader is the “entry point” - but I can’t see where to “feed in” the my filtered text data…

Best regards,

Bernd

Load Data infile unable to load a file of type binary

2011-11-16T18:28:56Z

Hi,
I have a text file produced in windows that i am trying to load into infobright using the loader.
Its a tab-delimited text file of type binary(when i run file -i filename ) i get this answer. When i try to load it using the loader it fails and mentions wrong type of column definition. although all columns in the table are of type varchar.
Could someone help me with this
infobright has been installed in red hat linux os

Regards
Sumit

High CPU for several hrs on LOAD DATA INFILE

2011-11-01T17:31:47Z

I have a data load which is consuming 99% CPU for several hours now. Cannot determine what the underlying issue is. Is this usually an indicator of a particular problem. I have not run into this problem prior to creating new extract files based on a load of new data to the source system. These are pipe-delimited files. I have confirmed layout is correct.

Parallel Load

2011-10-29T00:20:27Z

How to built parallel load (Distributed Load Processing) on infobright ?.. i need faster….
Because I have loaded 100 mio records with record length about 305 byte/record and conducted using single load, need arround 37 minutes, as below:
My Server: Sun xFire x2200 with 8GB RAM

mysql> set @bh_dataformat = ‘txt_variable’;

mysql> load data infile ‘/export/home/mscs.100mio.txt’ into table msc fields terminated by ‘|’;
Query OK, 100000000 rows affected (37 min 30.84 sec)
Records: 100000000 Deleted: 0 Skipped: 0 Warnings: 0

please advise..

BR
Sriwisnu

Infobright 4.0.4 and talend on Windows server 2008 R2

2011-10-03T11:23:30Z

last week we spent an inordinate amount of time trying to fix an issue with Talend and IB 4.0.4 on a Win 2008 R2 Server. we could not get the IB fast loader to load data in Talend. we kept getting an access denied message for the named pipes. long story short we found the solution

Something we did not know was that even though We were set up as administrators on the box you must tell Windows Server 2008 explicitly to run the application (Talend) as the Administrator. You can do this as a one time thing by right clicking on the launch icon for Talend and clicking “run as administrator”. (you will get a dialog box asking if you are really sure you want to do this)

if you want to always run the application as an administrator then right click on the launch icon go to properties. On the dialog box that opens, click on the compatibility tab and at the bottom, check the “Run this program as an administrator” box. even when you run it this way, you will get the “are you really sure” box.

Jon Finerfrock