Bug #1915
open(GIT) CSS Parser consumes enormous amounts of memory while compiling
0%
Description
While compiling WT under GCC 4.X (tested on 4.5, 4.7.3, and 4.8.0), the CSS Parser will rapidly exhaust all available virtual memory on a 32 bit system, and enormous and possibly unbounded amounts of virtual memory on a 64 bit system. This seems to be caused by the (new?) spirit based CSS parser.
Simply disabling spirit with WT_NO_SPIRIT solves the issue, but obviously you are using the (fantastic) spirit framework for a reason. It appears to be that the template metaprogramming machinery is not hitting an upper recursion limit (which would cause a halting of compilation), but is consuming exponentially more states with each recursion. G is slated to incorporate a number of optimizations which will change this from the current exponential cost into a much more acceptable linear cost through a state memorization mechanism, but it doesn't appear to be integrated in 4.7, which is the current stable branch. Maybe you are using 4.8 with this optimization?
I'm not sure if this is a bug or not (I can't compile Wt to test the functionality without disabling spirit), but it's certainly a compatibility concern.
Updated by Wim Dumon almost 11 years ago
http://boost-spirit.com/home/articles/doc-addendum/faq/#memory
Can you check if this is true? Maybe we should check if we can override the debug settings for this file only for practical reasons.
Wim.
Updated by Alice Margatroid almost 11 years ago
Even with this setting, it appears to consume exponential amounts of memory and quickly overwhelms my more than modest machine. I even tried on GCC 4.8, to no avail.
Updated by Koen Deforche almost 11 years ago
Hey,
We are using gcc 4.6.3 and 4.7.2 to build. The compiler takes up to 2 GB, so that's indeed pretty wild... (debug build)
I'm unsure if we should drop spirit for this reason as compilers have indeed announced that they will improve.
Regards,
koen
Updated by Alice Margatroid almost 11 years ago
Hey,
I've used 4.7.2 and the new 4.8 and I am still getting pretty obscene memory usage (2GB to 4GB doesn't seem uncommon).
My concern is mostly that on a 32 bit computer (they are still around, I know), you may easily run out of mappable virtual memory without the page table expansion support (PAE), which windows doesn't (for licensing reasons). This means with the new CSS Parser, a wide variety of people will be simply unable to compile without disabling the parser's use of spirit.
I'll investigate to see if your particular parser might be able to be optimized.
Alice