Joinutility seperatorLogin utility separator Infobright.com
   
4 of 4
4
load data too slowly
Posted: 01 June 2009 06:46 AM   Ignore ]   [ # 46 ]  
Jr. Member
RankRank
Total Posts:  87
Joined  2008-08-18

Hi Thanh,

Thank you very much for your response.
From outputs from gdb I can see that you have hit a bug in IB joining algorithm. I will create corresponding ticket in Infobright bug tracking system soon. Could you please confirm that when server hang it was probably executing the following query:

SELECT banneridB.locationIDSUM(totalClick)totalClick 
FROM 
    
SELECT banneridipCount(guid)`totalClick
    
FROM ad0509 WHERE clickorview=1  
    
AND timecreate BETWEEN '2009-05-13 16:00:00' AND '2009-05-13 16:29:59' 
    
GROUP BY banneridip 
    
) AS A  
LEFT JOIN 
`ip2location_bh` AS B on A.ip B.`ip`  
GROUP BY banneridB.locationID

As about server hangs that you have encountered. This is normal when you run program under gdb and it hit some problem. This is exactly what happend in your case and thanks to that we know where we should look for a bug in ICE source code.

Unfortunately we still don’t know why you can’t load new data to ad0509 table. This is just a guess but it is possible that this is an implication of problem with above query. However we should confirm reject this hypothesis.
It looks that the problem that you hit when you scheduled just data loads was caused by the fact that the server was doing nothing for a long time. This may happen when you run program under gdb.

To be continued…

Profile
 
Posted: 01 June 2009 01:11 PM   Ignore ]   [ # 47 ]  
Newbie
Rank
Total Posts:  42
Joined  2009-03-26

Hi Michal,

I want to notice you that I have another server (It’s called Slave server with only 16Gb of RAM and 8 cores in CPU : like a backup server, It’s running the latest of ICE on Linux).
This server and the “problem server” above were running with the same tasks (scheduled to load new data into tables with the same files,  and the same query (include the join query above)). And I don’t know why the slave server never crashes while the “problem server” with 48Gb of RAM and 16 cores in CPU always crashes. I wonder that may be the hardware is the main reason. Is it right?

Signature 

Thanh Nguyen

skype: thanhntvt
Website: Admicro.vn

Profile
 
Posted: 02 June 2009 09:12 AM   Ignore ]   [ # 48 ]  
Newbie
Avatar
Rank
Total Posts:  12
Joined  2008-08-18

Hello.

I have prepared gdb script that can log process state information
that can be helpful for us to diagnose problem.

I have attached this script (gzip compressed) in this very post.

gdb -/path/to/gdb-dump-meta-state.gdb --args /usr/local/infobright/bin/mysqld_debug --defaults-file=/etc/my-ib.cnf --basedir=/usr/local/infobright --datadir=/usr/local/infobright_home/data --user=mysql --log-queries-not-using-indexes --pid-file=/usr/local/infobright_home/data/localhost.pid --log-error=/usr/local/infobright_home/data/bh.err --socket=/tmp/mysql-ib.sock--port=3306 --gdb 

where /path/to/gdb-dump-meta-state.gdb is path to downloaded file.

After you run command I have specified above you should run

run 

gdb command.

When process crashed (gdb prompt will be available again) you shall
enter at gdb prompt following command:

dump-meta-state meta-state.log 

where meta-state.log is path to file where state data should be stored.

Next find aforementioned log file and sent id to us.

Good luck.

[ Edited: 04 June 2009 02:35 AM by Marcin Konarski]
File Attachments 
gdb-dump-meta-state.gdb.gz  (File Size: 1KB - Downloads: 221)
Profile
 
Posted: 05 June 2009 12:42 AM   Ignore ]   [ # 49 ]  
Newbie
Rank
Total Posts:  42
Joined  2009-03-26

Hi Michal and Marcin,

I’m sorry for very late reply.

About the Marcin’s script: Yesterday, I have run it and let me talk about the 1st crashes.
1st crashes: after about 30 min running, I tried to test some of queries and all things were fine. But crashes after I tried to kill an idle connection (an idle state, not querying and not load new data) and gdb threw the 1st log (log name: meta-state.log )
2nd crashes: after the 1st crashes, I restarted ICE and run script again. after this step I run my service that scheduled to load new data into the ad0509 table every 1 minutes, and query every 30 minutes (and I used a flag to make sure the task loading new data and querying can’t run in the same time). But ICE crashes after 3 hours of running time. Gdb threw the 2nd log (log name: meta-state2.log). And I did not restart the ICE to test again…

I have attached the log files in this post. I Hope you can fixed the problem soon. I’m looking forward to hearing from you.

PS: as I said in previous post that I have 2 servers running with the same task, but one of the servers was very stable while the another one always crashes after a few hours of running time. for more I have attached the “.dmsg files” both of servers.
stable server: 100.dmsg
“problem server”: 126.dmsg

Thanks.

File Attachments 
meta-state.zip  (File Size: 3KB - Downloads: 261)
dmsg log.zip  (File Size: 17KB - Downloads: 264)
Signature 

Thanh Nguyen

skype: thanhntvt
Website: Admicro.vn

Profile
 
   
4 of 4
4