Feature #7675
closedAvoid Huge Heap Allocation in CgiParser for File Uploads
0%
Description
In the CgiParser::parse function a buffer is allocated to hold the entire request. The request is then read into the buffer.
https://github.com/emweb/wt/blob/4.4.0/src/web/CgiParser.C#L212
We were looking to use Wt to handle file uploads on devices that only have 256 MiB of RAM. Allocating a buffer on the heap for a 100 MiB file upload would cripple the device.
Could the implementation change to support streaming the file upload to the spool file on the disk in chunks of 8 KiB (or some other reasonable number)? It might be nice to split file upload and non-file upload requests into different code paths, and have different configuration options for each. This would allow for non-file upload requests to continue to allocate a buffer limited by the MaxRequestSize configuration and to read everything in all at once. But also allow file uploads to have a separate configuration option, MaxFileUploadRequestSize, that could be much larger, because it isn't allocated on the heap.
Updated by Roel Standaert over 3 years ago
- Status changed from New to Feedback
Is this something you actually observed occurring and identified as an issue?
You are talking about uploads, which should not be handled by that code. Uploads are handled differently, and are limited by max-request-size, configurable in wt_config.xml. They should normally be spooled to a file (don't know off the top of my head how large the chunk size is).
That code should normally only handle form data, which, is limited by MIN(max-request-size, max-formdata-size).
Updated by Wim Dumon over 3 years ago
Also note the configuration parameter max-memory-request-size, which will cause file uploads to be buffered on disk.
Wim.
Updated by Aaron Wright over 3 years ago
I'm sorry. I made some wrong assumptions and didn't look closely enough at the conditionals and scoping. Thanks for reply.
Updated by Roel Standaert over 3 years ago
- Status changed from Feedback to Closed