forums.smartfoxserver.com

Hi All,

Month ago we have started our first MMO game for smartphone platforms (Android/iOS). We use SFS2X version 2.3.0 (unlimited users license) and Unity/.Net/Mono C# API client version 1.0.5. We noticed that at scale 7k ccu SFS is throttling, while below this value everything is ok. Typical symptoms are: sfs console (dashboard) is stuck for a moment, there are peaks on internet traffic (downs and ups – we observe that in monitoring of our server) – that suggests that sfs is blocked for a moment and traffic is lower for a few seconds, one moment later bigger traffic is back (sfs continues to work). I think that it is not a matter of lack CPU or heap memory. We use EC2 instance with 8 CPU and RightScale service to manage the instances and for monitoring, and we see that max cpu usage (at peak) is about 40%. There are no queues at extension pool thread or system pool thread. I don`t think it is a network problem because throttling occurs only above 7k ccu (we observe this everyday when peak of users comes). Thread dump also looks ok. We started even measuring GC (verbosegc command-line option) and FGC at peak is about 2,4secs and max GC is about 0,04secs). One more thing we have noticed is that, when the scale of ccu is below 7k ccu (SFS works like a charm) we observe (in the logs) that sessions are created smoothly (session creation is evenly distributed in time), but above 7k ccu (in our case) sessions are created in bigger packages – packages have about 100 sessions and they are created within 4 milliseconds. It looks like SFS buffers creation of the session and fire it in constant period of time (in our case every 5 seconds). This could be cause of throttling of the whole server, or this is only a result of something else. Lapo, what do you think? Is there any parameter that we could/should tune? Does anyone have the same problem?

Thanks in Advance,
Robert

Hi,
it would be good to update the server to the latest 2.5 version. There have been several improvements that might help. Also the most recent AdminTool Dashboard provides clearer history of queues and memory status.

There could be several reason for the symptoms you have reported and I think you have already mentioned them.
-- GC, definitely could be part of the issue but you seem to have checked into it already.
-- Server queues. This would be easier to inspect with the latest update.

As regards the bursts of connection in a few ms. that sounds unfamiliar and a little suspicious. Maybe you're receiving DDOS attacks but you don't notice it until you reach a certain load? Difficult to say from here. By checking the logs you should be able to see the IP addresses of those clients.

Another possible tool for further analysis is the VisualVM cpu sampler which can show you at runtime which threads are working the most.

Hi Lapo,

Thanks for your replay. Indeed this burst of session creation is strange, but each session has different IP address so DDOS attack is unlikely. If you would have any more hints please write me.

Thanks,

Yes, I agree.
I would definitely go ahead and update the server and the check the server queues.
Also keep an eye on log files for anomalous errors. If none of this helps the VisualVM inspection should shed more light.

Thanks

Hi Lapo,

Indeed we get deeper into VisualVM inspection and found out that that SocketReader thread is getting more and more busy while the scale of the ccu is getting higher. On the scale about 7000 ccu SocketReader thread spends 80% on running state. So we did CPU sampling and noticed that 75% of execution time takes SmartFoxServer.onSessionClosed() method call. Quick decompilation of this class showed us that inside this method there is a debug log: log.debug(userManager.getAllUsers().toString(). This call takes 71% of time. It looks like building this string takes a while and blocks SocketReader thread (on 7k ccu scale) so that new sessions are not created on real time but on bigger bunches. Even, if we have log level set on WARN, this method is called and string is built, but log is not routed to the appender. What do you think Lapo? Does it make sense for you?

Regards,
Robert

Hi Robert,
please take into account that decompilation goes against the license agreement accompanying the software.
There's no need or requirement to decompile the code for reporting an issue. Hope it is clear.

I have added a note in the bug-database for high priority, so that it is addressed in the next release.

Thanks