Monitoring queues

hasbean · Post by **hasbean** » 19 May 2011, 06:18

I've had my game in production for the past two months now, and so far so good. However, I've been seeing a problem that I can't seem to put my finger on. At random times, I seem to lose a whole bunch of users. When I check the logs, I find that the users complain the server is unresponsive. After about a minute later, the game returns to normal.

Graph shows a big one at 00:00

I'm wondering if it's the Extension Controller thread choking on something. I only have one Extension thread running since concurrency will be the death of me. I'm not sure why the game returns to normal after a minute or so since the Extension thread doesn't access anything external other than the database, and I'm pretty sure the database isn't the problem.

Now, I'd like to find some way to have a thread monitor the Extension Controller queue and see if the queue size is huge. That way, I might be able to pinpoint what caused the issue.

Help is much appreciated

Post by **Lapo** » 19 May 2011, 14:59

On the contrary I would say that the DB could really be the problem. A call to the DB slows down the thread very much.
Typically an extension call without DB code executes in microseconds (1millionth of a sec) while a DB call takes milliseconds (3 orders of magnitude)
If you get a bit of high concurrency you'll get the temporary slowdown.

The queues can be monitored from our AdminTool if you run RC2a.
Another thing to check is the amount of dropped messages.

hasbean · Post by **hasbean** » 23 May 2011, 16:45

I've added dbmon.pl to monitor the mysql db, and queries all finish normally without slowdowns. (0.026s)

I also notice that the Extension thread is actually still running fine during this user drop.

I've added a graph to check the dropped packet rate using SmartFoxServer.getInstance().getStatsManager().getTotalOutgoingDroppedPackets()

I notice dropped packets jump during the user drop. I'm pretty sure it's not a network problem either since I have an SFS 1.6 server running right next to it with no user drops in the graph.

I can't figure out how to graph out the queues since I can't find the method that exposes those numbers.

I'm stumped. I'm not really sure what else to check. I wish I could predict when it would happen, but it seems to be completely random

tchen · Post by **tchen** » 26 May 2011, 19:08

Try monitoring the GC as well.