Project

General

Profile

Support #7749

Crash (SEGV) when using WFileUpload within WQApplication

Added by Chris Sykes 17 days ago. Updated 5 days ago.

Status:
New
Priority:
High
Assignee:
-
Target version:
-
Start date:
10/06/2020
Due date:
% Done:

0%

Estimated time:

Description

Description

We've been seeing crashes when uploading larger files with WFileUpload
from within a WQApplication (as per the wtwithqt example).

We first saw these on Wt 3.3.8, and they still occur on Wt 3.7.0.

It looks to me like the issue is a data-race / thread synchronisation problem.
More details are provided below, and I've attached a minimal project that
reproduces the crash for me.

Issue #6548 contains stack-traces of the crash when I first noticed this on Wt 3.3.8.

Environment

  • Fedora 31 x86_64
  • Wt 3.7.0
  • Qt 5.13.2

Steps to reproduce the crash

  1. Unpack the attached example project.
  2. Build the code

    $ mkdir build && cd build
    $ cmake ..
    $ make -j 4

  3. Start the example server

    $ cd .. # out of build dir, back into source.
    $ ./run build/upload-recreate

  4. In the browser window, click "Browse" and select a
    large file (e.g. > 100 MiB). I see the crash on
    most uploads.

App console output:

[2020-Oct-06 16:30:36.646810] 1633202 - [debug] "WQApplication: [thread] handling event"
[2020-Oct-06 16:30:36.646820] 1633202 - [debug] "WQApplication: [thread] done handling event"
[2020-Oct-06 16:30:36.646824] 1633202 - [debug] "WQApplication: [thread] signaling event done"
[2020-Oct-06 16:30:36.646832] 1633202 [/ oRm722jntsoq5tnZ] [debug] "WQApplication: notifying thread"
[2020-Oct-06 16:30:36.646845] 1633202 - [debug] "WQApplication: [thread] handling event"
[2020-Oct-06 16:30:36.646854] 1633202 - [debug] "WQApplication: [thread] done handling event"
[2020-Oct-06 16:30:36.646858] 1633202 - [debug] "WQApplication: [thread] signaling event done"
[2020-Oct-06 16:30:36.654864] 1633202 [/ oRm722jntsoq5tnZ] [debug] "WQApplication: notifying thread"
[2020-Oct-06 16:30:36.654941] 1633202 - [debug] "WQApplication: [thread] handling event"
./run: line 14: 1633202 Segmentation fault      (core dumped) "$APP" --docroot="$PWD;/resources" --http-address "$ADDR" --http-port "$PORT"

gdb stack trace:

(gdb) info threads
  Id   Target Id                                             Frame 
  1    Thread 0x7ffff3f7fc80 (LWP 1633820) "upload-recreate" 0x00007ffff6728412 in sigtimedwait() from /lib64/libc.so.6
  2    Thread 0x7fffe6f86700 (LWP 1633821) "upload-recreate" 0x00007ffff7a1fd45 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  3    Thread 0x7fffe6785700 (LWP 1633822) "upload-recreate" 0x00007ffff7a23610 in __lll_lock_wait () from /lib64/libpthread.so.0
  4    Thread 0x7fffe5f84700 (LWP 1633823) "upload-recreate" 0x00007ffff7a1fd45 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  5    Thread 0x7fffe5783700 (LWP 1633824) "upload-recreate" 0x00007ffff67ec9fe in epoll_wait() from /lib64/libc.so.6
* 6    Thread 0x7fffe4f82700 (LWP 1633835) "QThread"         std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Wt::WResource*>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Wt::WResource*> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Wt::WResource*> > >::_M_lower_bound (this=<optimized out>, 
    __k=<error reading variable: Cannot access memory at address 0x7fffe45bf018>, 
    __y=0x7fffe000c3a8, __x=0x7fffd4005f90) at /usr/include/c++/9/bits/stl_tree.h:1945



(gdb) bt
#0  std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Wt::WResource*>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Wt::WResource*> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Wt::WResource*> > >::_M_lower_bound (this=<optimized out>,
    __k=<error reading variable: Cannot access memory at address 0x7fffe45bf018>,
    __y=0x7fffe000c3a8, __x=0x7fffd4005f90) at /usr/include/c++/9/bits/stl_tree.h:1945
#1  std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Wt::WResource*>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Wt::WResource*> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Wt::WResource*> > >::find (__k=<error reading variable: Cannot access memory at address 0x7fffe45bf018>,
    this=0x7fffe000c3a0) at /usr/include/c++/9/bits/stl_tree.h:2570
#2  std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, Wt::WResource*, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Wt::WResource*> > >::find (
    __x=<error reading variable: Cannot access memory at address 0x7fffe45bf018>,
    this=0x7fffe000c3a0) at /usr/include/c++/9/bits/stl_map.h:1194
#3  Wt::WApplication::decodeExposedResource (this=this@entry=0x7fffe000be30,
    resourceKey=<error reading variable: Cannot access memory at address 0x7fffe45bf018>)
    at ../src/Wt/WApplication.C:886
#4  0x00007ffff7516a0c in Wt::WebController::updateResourceProgress (this=<optimized out>,
    request=0x7fffe00070a0, current=90946977, total=342288591)  
    at ../src/web/WebController.C:474
#5  0x00007ffff7528dfc in boost::function0<void>::operator() (this=<optimized out>)
    at /usr/include/boost/function/function_template.hpp:677
#6  Wt::WebSession::notify (this=0x7fffe00044a0, event=...) at ../src/web/WebSession.C:2240
#7  0x0000000000489e23 in Wt::WQApplication::realNotify(Wt::WEvent const&) ()
#8  0x000000000048661f in Wt::DispatchThread::doEvent() ()
#9  0x0000000000485f20 in Wt::DispatchObject::onEvent() ()
#10 0x000000000046c945 in Wt::DispatchObject::qt_static_metacall(QObject*, QMetaObject::Call, int, void**) ()
#11 0x00007ffff7d007fa in QObject::event(QEvent*) () from /lib64/libQt5Core.so.5
#12 0x00007ffff7cd53b5 in doNotify(QObject*, QEvent*) () from /lib64/libQt5Core.so.5
#13 0x00007ffff7cd5448 in QCoreApplication::notifyInternal2(QObject*, QEvent*) ()

Files

upload-recreate.tar.gz (8.07 KB) upload-recreate.tar.gz Minimal project for reproduction Chris Sykes, 10/06/2020 05:57 PM
#1

Updated by Chris Sykes 17 days ago

I believe this is a data race/synchronisation issue because it does not reproduce every time (perhaps 75% of the time using a 135 MiB file).

Also, I'm unable to reproduce the crash if I change the thread pool size from 4 to 1 in main.cpp:

   99         server->ioService().setThreadCount(1);

The other visible difference after making that change is that the upload progress % shown in the web app now updates during the upload.

With the thread pool count set at 4, the upload progress stays at 0% throughout the upload, then jumps to "upload complete".

#2

Updated by Chris Sykes 5 days ago

I've not (yet) been able to reproduce the crash when using a WApplication rather than a WQAppliction,
so the issue is either caused by or significantly worsened by the WtQt integration.

When using WQApplication, the crash reproduces with the DispatchThread running the Qt event loop AND
also when running DispatchThread::myExec().

Also available in: Atom PDF