Our infobright 3.3.1 has crashed yesterday and since we’ve been rebooted him it all goes wrong…

bhloader keeps on loading even when the csv only 4 rows big.

no process is getting RES memory, only mysql takes up the 27GB VIRT memory (as defined in brighthouse.ini). Suddenly the box goes out of memory

when stracing bhloader i get:
...
stat(”/var/lib/mysql/facebox/LOG_MOBILE_PROFILEVIEW.bht/TA00000001797486.ctb”, 0x7ffff9a35730) = -1 ENOENT (No such file or directory)
stat(”/var/lib/mysql/facebox/LOG_MOBILE_PROFILEVIEW.bht/TA00000001797487.ctb”, 0x7ffff9a35730) = -1 ENOENT (No such file or directory)
stat(”/var/lib/mysql/facebox/LOG_MOBILE_PROFILEVIEW.bht/TA00000001797488.ctb”, 0x7ffff9a35730) = -1 ENOENT (No such file or directory)
stat(”/var/lib/mysql/facebox/LOG_MOBILE_PROFILEVIEW.bht/TA00000001797489.ctb”, 0x7ffff9a35730) = -1 ENOENT (No such file or directory)
stat(”/var/lib/mysql/facebox/LOG_MOBILE_PROFILEVIEW.bht/TA00000001797490.ctb”, 0x7ffff9a35730) = -1 ENOENT (No such file or directory)
stat(”/var/lib/mysql/facebox/LOG_MOBILE_PROFILEVIEW.bht/TA00000001797491.ctb”, 0x7ffff9a35730) = -1 ENOENT (No such file or directory)
stat(”/var/lib/mysql/facebox/LOG_MOBILE_PROFILEVIEW.bht/TA00000001797492.ctb”, 0x7ffff9a35730) = -1 ENOENT (No such file or directory)
stat(”/var/lib/mysql/facebox/LOG_MOBILE_PROFILEVIEW.bht/TA00000001797493.ctb”, 0x7ffff9a35730) = -1 ENOENT (No such file or directory)
stat(”/var/lib/mysql/facebox/LOG_MOBILE_PROFILEVIEW.bht/TA00000001797494.ctb”, 0x7ffff9a35730) = -1 ENOENT (No such file or directory)
stat(”/var/lib/mysql/facebox/LOG_MOBILE_PROFILEVIEW.bht/TA00000001797495.ctb”, 0x7ffff9a35730) = -1 ENOENT (No such file or directory)
stat(”/var/lib/mysql/facebox/LOG_MOBILE_PROFILEVIEW.bht/TA00000001797496.ctb”, 0x7ffff9a35730) = -1 ENOENT (No such file or directory)
stat(”/var/lib/mysql/facebox/LOG_MOBILE_PROFILEVIEW.bht/TA00000001797497.ctb”, 0x7ffff9a35730) = -1 ENOENT (No such file or directory)
stat(”/var/lib/mysql/facebox/LOG_MOBILE_PROFILEVIEW.bht/TA00000001797498.ctb”, 0x7ffff9a35730) = -1 ENOENT (No such file or directory)
stat(”/var/lib/mysql/facebox/LOG_MOBILE_PROFILEVIEW.bht/TA00000001797499.ctb”, 0x7ffff9a35730) = -1 ENOENT (No such file or directory)
...

There must be some corruption somehwere, can anybody help me?

Hi !

Could you check manually if the files, e.g. /var/lib/mysql/facebox/LOG_MOBILE_PROFILEVIEW.bht/TA00000001797493.ctb exist? Try
ls -l /var/lib/mysql/facebox/LOG_MOBILE_PROFILEVIEW.bht/

wc /var/lib/mysql/facebox/LOG_MOBILE_PROFILEVIEW.bht/TA00000001797493.ctb

Correct, I forgot to add that information.

Those files do not exist. The bhloader process is really looping through millions of those files until virtual memory is full and the box goes out of memory

root@warehouse01:/var/lib/mysql/facebox/LOG_MOBILE_PROFILEVIEW.bht# ls
ab_switch         TA00001000000000.ctb TA00002000000001.ctb TA00003.bck       TA00004.ctb       TA00005DPN.ctb     Table.ctb   TB00005.ctb
TA00000000000000.ctb TA00001000000001.ctb TA00002.bck       TA00003.ctb       TA00004DPN.ctb     TA00006000000000.ctb TB00000.ctb TB00006.ctb
TA00000000000001.ctb TA00001.bck       TA00002.ctb       TA00003DPN.ctb     TA00005000000000.ctb TA00006000000001.ctb TB00001.ctb
TA00000.bck       TA00001.ctb       TA00002DPN.ctb     TA00004000000000.ctb TA00005000000001.ctb TA00006.bck       TB00002.ctb
TA00000.ctb       TA00001DPN.ctb     TA00003000000000.ctb TA00004000000001.ctb TA00005.bck       TA00006.ctb       TB00003.ctb
TA00000DPN.ctb     TA00002000000000.ctb TA00003000000001.ctb TA00004.bck       TA00005.ctb       TA00006DPN.ctb     TB00004.ctb

Hi!

I guess it is alright, just to check - what are the permissions to /var/lib/mysql/facebox/LOG_MOBILE_PROFILEVIEW.bht and what is the user account to run ICE/bhloader ?

Your folder looks almost ok - 7 columns with small number of rows. However the presence of .bck files suggests, that the contents of the files got damaged.

What was the nature of the crash?

Yes, permissions are correct, everything is owned mysql and runs as mysql

The box ran out of memory, caused by another process next to mysql/bhloader]

brighthouse.log tells me:
2010-03-18 09:32:14 Error in attribute file: bad format
2010-03-18 09:32:14 Error in attribute file: bad format
2010-03-18 09:32:14 Error in attribute file: bad format
2010-03-18 09:32:14 Error in attribute file: bad format
2010-03-18 09:32:14 Error in attribute file: bad format
2010-03-18 09:32:14 Error in attribute file: bad format
2010-03-18 09:32:14 Error in attribute file: bad format
2010-03-18 09:32:14 Error in attribute file: bad format
2010-03-18 09:32:14 Error in attribute file: bad format
...
2010-03-18 09:32:15 2010-03-18 09:32:14 Error in attribute file: bad format
2010-03-18 09:32:15 2010-03-18 09:32:14 Error in attribute file: bad format
2010-03-18 09:32:15 2010-03-18 09:32:14 Error in attribute file: bad format
2010-03-18 09:32:15 2010-03-18 09:32:14 Error in attribute file: bad format
2010-03-18 09:32:15 2010-03-18 09:32:14 Error in attribute file: bad format
2010-03-18 09:32:15 2010-03-18 09:32:14 Error in attribute file: bad format
2010-03-18 09:32:15 2010-03-18 09:32:14 Error in attribute file: bad format
2010-03-18 09:32:15 2010-03-18 09:32:14 Error in attribute file: bad format
2010-03-18 09:32:15 2010-03-18 09:32:14 Error in attribute file: bad format
2010-03-18 09:32:15 2010-03-18 09:32:14 Error in attribute file: bad format
2010-03-18 09:32:15 The Infobright storage engine has encountered an unexpected error. The current transaction has been rolled back.
...
2010-03-18 09:32:15 Error: Unknown error.
2010-03-18 09:32:15 Error in attribute file: bad format
2010-03-18 09:32:15 Error in attribute file: bad format
2010-03-18 09:32:15 Error in attribute file: bad format
2010-03-18 09:32:15 Error in attribute file: bad format
2010-03-18 09:32:15 Error in attribute file: bad format
2010-03-18 09:32:15 Error in attribute file: bad format
...

Unfortunately, it looks the data files are damaged. Restore from backup or table recreation and reload can help.

Still I cannot see why the files got damaged. Apparently files for all the columns. Other tables are fine? Could you show
ls -l /var/lib/mysql/facebox/LOG_MOBILE_PROFILEVIEW.bht/

and attach e.g. TA00000.bck and TA00000.ctb

root@warehouse01:/var/lib/mysql/facebox/LOG_MOBILE_PROFILEVIEW.bht# ls -l /var/lib/mysql/facebox/LOG_MOBILE_PROFILEVIEW.bht/
total 1332620
-rw-rw——1 mysql mysql 377713961 2010-03-18 09:32 TA00000000000000.ctb
-rw-rw——1 mysql mysql 369761945 2010-03-16 16:54 TA00000000000001.ctb
-rw-rw——1 mysql mysql       0 2010-03-16 16:54 TA00000.bck
-rw-rw——1 mysql mysql     119 2010-03-18 09:33 TA00000.ctb
-rw-rw——1 mysql mysql   103304 2010-03-18 09:32 TA00000DPN.ctb
-rw-rw——1 mysql mysql 91930876 2010-03-18 09:33 TA00001000000000.ctb
-rw-rw——1 mysql mysql 89786582 2010-03-16 16:54 TA00001000000001.ctb
-rw-rw——1 mysql mysql       0 2010-03-16 16:54 TA00001.bck
-rw-rw——1 mysql mysql     119 2010-03-18 09:34 TA00001.ctb
-rw-rw——1 mysql mysql   103304 2010-03-18 09:33 TA00001DPN.ctb
-rw-rw——1 mysql mysql 44803272 2010-03-16 16:53 TA00002000000000.ctb
-rw-rw——1 mysql mysql 44222872 2010-03-16 16:54 TA00002000000001.ctb
-rw-rw——1 mysql mysql       0 2010-03-16 16:54 TA00002.bck
-rw-rw——1 mysql mysql     119 2010-03-18 09:34 TA00002.ctb
-rw-rw——1 mysql mysql   103304 2010-03-18 09:34 TA00002DPN.ctb
-rw-rw——1 mysql mysql 36707397 2010-03-16 16:53 TA00003000000000.ctb
-rw-rw——1 mysql mysql 36584211 2010-03-16 16:54 TA00003000000001.ctb
-rw-rw——1 mysql mysql       0 2010-03-16 16:54 TA00003.bck
-rw-rw——1 mysql mysql     119 2010-03-18 09:36 TA00003.ctb
-rw-rw——1 mysql mysql   103304 2010-03-18 09:34 TA00003DPN.ctb
-rw-rw——1 mysql mysql 41804541 2010-03-16 16:53 TA00004000000000.ctb
-rw-rw——1 mysql mysql 41695987 2010-03-16 16:54 TA00004000000001.ctb
-rw-rw——1 mysql mysql       0 2010-03-16 16:54 TA00004.bck
-rw-rw——1 mysql mysql     119 2010-03-18 09:37 TA00004.ctb
-rw-rw——1 mysql mysql   103304 2010-03-18 09:36 TA00004DPN.ctb
-rw-rw——1 mysql mysql 55835162 2010-03-16 16:53 TA00005000000000.ctb
-rw-rw——1 mysql mysql 54707265 2010-03-16 16:54 TA00005000000001.ctb
-rw-rw——1 mysql mysql       0 2010-03-16 16:54 TA00005.bck
-rw-rw——1 mysql mysql     119 2010-03-18 09:38 TA00005.ctb
-rw-rw——1 mysql mysql   103304 2010-03-18 09:37 TA00005DPN.ctb
-rw-rw——1 mysql mysql 38047708 2010-03-16 16:53 TA00006000000000.ctb
-rw-rw——1 mysql mysql 38173367 2010-03-16 16:54 TA00006000000001.ctb
-rw-rw——1 mysql mysql       0 2010-03-16 16:54 TA00006.bck
-rw-rw——1 mysql mysql     119 2010-03-18 09:39 TA00006.ctb
-rw-rw——1 mysql mysql   103304 2010-03-18 09:38 TA00006DPN.ctb
-rw-rw——1 mysql mysql     74 2009-11-23 10:01 Table.ctb
-rw-rw——1 mysql mysql     119 2010-03-17 16:05 TB00000.ctb
-rw-rw——1 mysql mysql     119 2010-03-17 16:07 TB00001.ctb
-rw-rw——1 mysql mysql     119 2010-03-17 16:08 TB00002.ctb
-rw-rw——1 mysql mysql     119 2010-03-17 16:09 TB00003.ctb
-rw-rw——1 mysql mysql     119 2010-03-17 16:10 TB00004.ctb
-rw-rw——1 mysql mysql     119 2010-03-17 16:11 TB00005.ctb
-rw-rw——1 mysql mysql     119 2010-03-17 16:13 TB00006.ctb
root@warehouse01:/var/lib/mysql/facebox/LOG_MOBILE_PROFILEVIEW.bht#

both files are in the tarball.

A quick question - the listing shows all .bck files are 0 bytes. But the file in tar.gz is 205 bytes. ??

Also ab_switch was listed at first, but not later…

Can you try
select count(*) from LOG_MOBILE_PROFILEVIEW;
Select * from LOG_MOBILE_PROFILEVIEW;

If not successful:

touch LOG_MOBILE_PROFILEVIEW/ab_switch
select count(*) from LOG_MOBILE_PROFILEVIEW;
Select * from LOG_MOBILE_PROFILEVIEW;

Basically, the .bck file were supposed to be valid column headers. But they are mangled completely - like random content. It was recognized as an old format, and new .ctb files were created, unfortunately containing wrong (very large) number of rows. The heavy VM usage by the server is caused by preparation of data structures for these large number of rows, I guess.

your explanation sounds plausible!

mysql> select count(*) from LOG_MOBILE_PROFILEVIEW;
+—————+
| count(*) |
+—————+
|  63347 |
+—————+
1 row in set (0.00 sec)

mysql> Select * from LOG_MOBILE_PROFILEVIEW;
ERROR 1105 (HY000): Brighthouse other specific error: BH result type not convertible to MySQL type

touch LOG_MOBILE_PROFILEVIEW.bht/ab_switch

mysql> select count(*) from LOG_MOBILE_PROFILEVIEW;
+—————+
| count(*) |
+—————+
|  63347 |
+—————+
1 row in set (0.00 sec)

mysql> Select * from LOG_MOBILE_PROFILEVIEW;
ERROR 1105 (HY000): Brighthouse other specific error: BH result type not convertible to MySQL type
mysql> show warnings;
+———-+———+————————————————————————————————————————+
| Level | Code | Message                                                 |
+———-+———+————————————————————————————————————————+
| Error | 1105 | Brighthouse other specific error: BH result type not convertible to MySQL type |
| Error | 1105 | Unknown error                                             |
+———-+———+————————————————————————————————————————+
2 rows in set (0.00 sec)

Can this be fixed by ignoring the broken files? Eventually having some data loss.

Hi !
Data reload or restore from backup should help.