Project

General

Profile

Wt website downtime

Added by kevin bailey almost 6 years ago

Hi all! So recently I have been having some strange issues with my Wt based website. I am having a hard time determining if it is a Wt issue or something else.

I noticed that sometimes after several hours of running the website it will just stop responding entirely. When I log into the server the Wt site is hosted on my Wt executable is still running. I set the following parameters on my Wt executable as well ---errroot, and ---accesslog. I noticed that errroot didn't do anything during these time periods but the accesslog is constantly scrolling through these strange POST requests over and over again, even just for one session. e.g.

ip.add.r.es - - [2018-Jul-24 09:03:47.526397] "POST /?wtd=aRlZsi1hEdGRn5Ic HTTP/1.1" 200 88
ip.add.r.es - - [2018-Jul-24 09:03:47.608348] "POST /?wtd=aRlZsi1hEdGRn5Ic HTTP/1.1" 200 88
ip.add.r.es - - [2018-Jul-24 09:03:47.690601] "POST /?wtd=aRlZsi1hEdGRn5Ic HTTP/1.1" 200 88
ip.add.r.es - - [2018-Jul-24 09:03:47.772498] "POST /?wtd=aRlZsi1hEdGRn5Ic HTTP/1.1" 200 88
ip.add.r.es - - [2018-Jul-24 09:03:47.860816] "POST /?wtd=aRlZsi1hEdGRn5Ic HTTP/1.1" 200 88

Is this normal? I will also note that after restarting the Wt executable the site comes back online like normal. Furthermore there is no excessive CPU or RAM usage on the server during these outages. And finally the traffic to the site is also pretty low, maybe 20 visitors a day.

The website also will crash completely and throw the following errors on occasion:

Error occurred. MySQL: Failed to connect to database server: Can't create UNIX socket (24)

Error occurred. MySQL: Failed to connect to database server: Too many connections

I noticed that when these errors occur mariadb is showing many many sleeping processes under 'show processlist;'. On killing these processes/restarting madiadb the website does continue to work appropriately. I am also fairly confident that I have no other major processes running on the server interacting with that mariadb database so I am fairly confident that Wt is causing this issue...

The only change I have made to /etc/wt/wt_config.conf is the following:

<progressive-bootstrap>true</progressive-bootstrap>

Finally one other strange thing I noticed was that the 'auth_token' table is filled with enties that date back to 2017-05-21 11:40:44... Is this normal? Could this be causing this issue?

Any help would be greatly appreciated! Please let me know if anymore information is required.


Replies (2)

RE: Wt website downtime - Added by Wim Dumon almost 6 years ago

Hey,

It looks like you're running out of available file descriptors. Can you use 'lsof' on your process and check if something seems off/weird? The progressive-bootstrap feature on itself should not be causing this and I don't see an immediate reason as to why it could trigger it.

BR,

Wim.

RE: Wt website downtime - Added by lm at almost 6 years ago

I wouldn't think you're running out of file descriptors, but you may be running out of sockets? You only have 65k TCP ports available (minus what are being used), but the number of file descriptors is much higher. ss -an; may help diagnose this. (Certainly run lsof; as well!)

    (1-2/2)