We are planning on using ICE to store and query approx. 10TB of server log data.  I have been testing load/query performance on a dev box, and am now being asked for specs on what will become the production hardware.  From what I have seen in my testing both loader and query performance with ICE appears to be limited by the speed of a single CPU core, which I assume comes from the compression and decompression of the data.  So CPU clock speed/performance is at the top of my requirement.  Also, I saw very little IO wait in my testing so disk access speed would not seem to be a priority.

My question is how will the amount of RAM available effect performance?  Will 16Gig provide significantly better performance than 8Gig?

Hi,

16 GB should be better than 8 GB, if proper memory settings are set in brighthouse.ini. For 16 GB I would set ServerMainHeapSize = 12000.

CPU performance (one core) is also important, as well as memory speed.

Regards,

Thanks for the reply Jakub.  Is there a point of diminishing return on the amount of RAM?  Would 32G offer further improvements over 16?  And could you explain how the increased RAM is used to improve performance?  Larger buffers to cache data in memory?

Increasing RAM will boost performance, at least within 32 GB range. Then the difference for one user will be smaller, i.e. 64 GB will be only slightly better than 32 GB.

More memory means larger heap size which may be set in brighthouse.ini. And the heap size influences query performance in two ways:

- Internal buffers for query execution algorithms are based on the heap size. E.g. setting 10000 instead of 5000 will allow joining to use twice larger hash buffer, which decreases a number of joining passes two times. So the joining will be about two times faster. In some cases the difference will be even larger, because we can fit a whole structure in memory and avoid multipass processing with disk caching. Such benefit we may observe also in ordering, grouping, finding distinct values etc.

- The internal heap not used for such buffers is utilized to keep uncompressed data packs. The larger memory available for data pack caching, the higher probability that we will not need to read and decompress data pack in a next query, or in later steps of the same query.

However, if data tables are small, and queries are not complicated, then we may not see any difference between 1 GB and 32 GB.

Regards,

Hi !

To add a little to what Jakub has said - if you will have a single user, the difference between 16 and 32 GB will be visible, but it will be even more visible for multiuser scenarios. Adding more than 32GB in the current release makes sense mainly for multiuser (many queries running in parallel).

Cheers,

Thanks alot guys, I will spec. the memory at 32G.