smartfox crash every day

[styx]

In my project, I watch the situation:
during peak hours:

- users connect to my zone at a speed of 1-10 per second,
- Rooms are created and removed an average of 1 in every 1s.
- logged-in users is from 5000-6000
- Outgoing Message Queue Size: 0-15k
- active threads 150-280
- The Handler Queue Size: 0
- Extension Handler Queue Size: 0
- Users write a lot of messages in chat

smartfox dying almost every day (nobody can connect, but those who are already logged work without a problem), we have to restart smartfox

[BigFIsh]

Did you get any server errors?

[styx]

There are no specific errors, only this:

from admin panel I have errors like

[ WARNING ] [id: 398449] (User.exitAllRooms): Problems during user removal: guest_1537338 (2 error per 1 minute)
[ WARNING ] [id: 11] (SystemHandler.handleLoginRequest): Connection was closed during login phase. (5 per 1 second from time to time)

from other logs:

Invalid connection = null
java.lang.NullPointerException
at it.gotoandplay.utils.net.xmlsocket.XMLSocket.send(XMLSocket.java:93)
at it.gotoandplay.smartfoxserver.httpbox.util.SFSHandshake.<init>(SFSHandshake.java:30)
at it.gotoandplay.smartfoxserver.httpbox.filter.SessionIDFilter.doFilter(SessionIDFilter.java:114)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1148)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:387)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:535)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:880)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:747)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:451)
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect(Native Method)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:81)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:65)
at org.xsocket.stream.io.impl.IoProvider.openSocket(IoProvider.java:346)
at org.xsocket.stream.io.impl.IoProvider.createClientIoHandler(IoProvider.java:209)
at org.xsocket.stream.Connection.<init>(Connection.java:160)
at org.xsocket.stream.NonBlockingConnection.<init>(NonBlockingConnection.java:394)
at org.xsocket.stream.NonBlockingConnection.<init>(NonBlockingConnection.java:209)
at it.gotoandplay.utils.net.xmlsocket.XMLSocket.connect(XMLSocket.java:54)
at it.gotoandplay.smartfoxserver.httpbox.util.SFSHandshake.<init>(SFSHandshake.java:29)
at it.gotoandplay.smartfoxserver.httpbox.filter.SessionIDFilter.doFilter(SessionIDFilter.java:114)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1148)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:387)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:535)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:880)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:747)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:451)

[styx]

today, after recent changes in the number of threads works quite well
only 42 active threads

[styx]

2011/02/15 23:37:04.276 - [ WARNING ] [id: 12] (SmartFoxServer.checkNewConnections): CheckNewConnection Problem: closed channel.

[Lapo]

Hi,
none of the above errors are very indicative of a bad issue.
What is the CPU consumption of the server process when you get the problem?
I would suggest to perform a thread dump when the problem arises.

The thread dump in the console can be done by hitting ctrl+break (Windows) or sending kill -SIGQUIT to the process (under Linux/Unix)

Additionally if you run Java6 you could use a visual tool such as VisualVM to analyze what's going on:

https://visualvm.dev.java.net
https://visualvm.dev.java.net/description.html

[styx]

Now I saw that when smartfox crash "system handler queue size" increased to infinity. In config.xml I have <SysHandlerThreads> 2 </ SysHandlerThreads>
possible that something in the log phase of crashing and new users can not log on. It gave the entire log phase asynchronous side PHP

Code: Select all

case _events.InternalEventObject.EVENT_LOGIN:
var aload = new LoadVars();
aload.onLoad = function(success, errorMsg) {					
if (success) {
//login user
}
  aload.load(url);
break;	

This code creates more threads but it is safer

And what do you think of other errors that I showed?

[melojoy]

Hi Lapo,
To use VisualVM we have to run on server jstatd utility.
Where is available for download jstatd? I have only this link: http://download.oracle.com/javase/6/doc ... statd.html

Thanks,
Maurizio

[Lapo]

Probably you need the JDK instead of the JRE?
Just download the latest JDK from Oracle's website.