Regards,
darnpunk
Messages per second
Messages per second
Is there a way to count how many messages on average is being sent per second with SFS? With version 1.5.5 PRO I get alot of dropped messages and with version 1.6.1 PRO I get alot of disconnections. In both cases, they seem to tell me I am sending too many messages per second. I need help/tips how to measure this. Can anyone help?
Regards,
darnpunk
Regards,
darnpunk
It doesn't depend much on SFS, it depends on your average network lag.
If the client pings the server and finds an average time of, say, 100ms you can easily calculate it:
In this case it would be 10 ( 1000 / 100 )
Usually a lag of 100ms is acceptable and there can be users who have higher lag values, so I wouldn't recommend sending more than 10-20 messages on each second. If you need more you probably should rethink how your update system works and make the client side smarter about how to deal with missing/lagging data.
If the client pings the server and finds an average time of, say, 100ms you can easily calculate it:
Code: Select all
MAX_MESSAGES = 1000 / PingTimeUsually a lag of 100ms is acceptable and there can be users who have higher lag values, so I wouldn't recommend sending more than 10-20 messages on each second. If you need more you probably should rethink how your update system works and make the client side smarter about how to deal with missing/lagging data.
Hi Lapo,
Thanks for the reply. In your formula, what does the value 1000 represent? I assume its 1000 = 1 second?
By the way, does version 1.6.1 and 1.5.5 have different policies on disconnecting users? I seem to experience mass disconnections on version 1.6.1 quite often.
I observe that if a particular client has dropped messages more than the specified value in config file <MaxAllowedDroppedPackets>, the console will display a message that says something like "Disconnecting user dropped packets too high, username, packets dropped". I understand that when that message shows up in the console, it is an intended disconnection by SFS as the user dropped more packets than allowed.
However, I face often mass disconnections like around 50 users at one go. I checked for a similar message as above but nothing showed up. I also checked the dropped packets and it did not increase. It happens during off peak hours too. I checked that the bandwidth is not fully used up and there was no lag experienced by me or other users.
As soon as I switched back to 1.5.5 the mass disconnection stopped. I use the same codes for both client and server side (extensions, game client etc). So I am wondering if anything is causing it?
Regards,
Arsyad
Thanks for the reply. In your formula, what does the value 1000 represent? I assume its 1000 = 1 second?
By the way, does version 1.6.1 and 1.5.5 have different policies on disconnecting users? I seem to experience mass disconnections on version 1.6.1 quite often.
I observe that if a particular client has dropped messages more than the specified value in config file <MaxAllowedDroppedPackets>, the console will display a message that says something like "Disconnecting user dropped packets too high, username, packets dropped". I understand that when that message shows up in the console, it is an intended disconnection by SFS as the user dropped more packets than allowed.
However, I face often mass disconnections like around 50 users at one go. I checked for a similar message as above but nothing showed up. I also checked the dropped packets and it did not increase. It happens during off peak hours too. I checked that the bandwidth is not fully used up and there was no lag experienced by me or other users.
As soon as I switched back to 1.5.5 the mass disconnection stopped. I use the same codes for both client and server side (extensions, game client etc). So I am wondering if anything is causing it?
Regards,
Arsyad
Correct. 1 second == 1000 millisecondsThanks for the reply. In your formula, what does the value 1000 represent? I assume its 1000 = 1 second?
Network lag is usually calculated in ms.
Yes, since 1.5.9 we introduced a more fine-grained system to adjust the tolerance of dropped messages. More infos are available in chapter 2.2 of the docs. If the default settings don't work well for you, you can customize them in order to meet your application requirements.By the way, does version 1.6.1 and 1.5.5 have different policies on disconnecting users? I seem to experience mass disconnections on version 1.6.1 quite often.
It usually involves doing a couple of tests in order to find the proper client queue size.
In the next releases we'll be adding even more fine grained tools, to control message priorities.
If there's no such message it's not depending on dropped messages. Maybe those are idle clients?However, I face often mass disconnections like around 50 users at one go. I checked for a similar message as above but nothing showed up.
This seems to confirm what I said before, they definitely sound like idle users. Make sure your maxIdleTime setting is not too strict.As soon as I switched back to 1.5.5 the mass disconnection stopped.
Additionally turn the console logging from INFO level to FINER level, this will show you more debug infos and you'll probably see what happens when you get multiple disconnections.
Hi Lapo,
My maxIdleTime is set to 600 seconds. The mass disconnection happens to active users too
. I set my maxIdleTime to 600 seconds in version 1.5.5 but this never happened. I allowed 20 dropped packets for each client but it does not help. Maybe I am sending/receiving too many bursts of messages? I'm going to set the console logging to FINER and see what I can catch from there.
My maxIdleTime is set to 600 seconds. The mass disconnection happens to active users too
. I set my maxIdleTime to 600 seconds in version 1.5.5 but this never happened. I allowed 20 dropped packets for each client but it does not help. Maybe I am sending/receiving too many bursts of messages? I'm going to set the console logging to FINER and see what I can catch from there.Hi again,
I am still facing mass disconnection. You can say a total disconnect of every user. The server doesnt crash however.
I am still using SFS 1.5.5 PRO as the mass disconnection occurs least for it. The network graph shows a healthy frequency and the bandwidth never reaches its peak. We have a 10mbps line and graph shows about approximately half is used up.
I'm suspecting there could be other issues. Is there anyway I can monitor this problem closely? Currently, I restarted the server and at 1 day, max connection of 69 users, the dropped out-msg shows 1.0K (from Admin Tool). This is not a good sign isn't it
Any help is appreciated.
Regards,
darnpunk
I am still facing mass disconnection. You can say a total disconnect of every user. The server doesnt crash however.
I am still using SFS 1.5.5 PRO as the mass disconnection occurs least for it. The network graph shows a healthy frequency and the bandwidth never reaches its peak. We have a 10mbps line and graph shows about approximately half is used up.
I'm suspecting there could be other issues. Is there anyway I can monitor this problem closely? Currently, I restarted the server and at 1 day, max connection of 69 users, the dropped out-msg shows 1.0K (from Admin Tool). This is not a good sign isn't it
Any help is appreciated.
Regards,
darnpunk
A cause of multiple disconnections could be too heavy messaging with wrong configuration of the user queues.
If clients receive lots of messages per second they might not always be able to keep up.
If this happens the server keeps data in the queue and waits the client to be able to receive more data.
You can configure the queue to allow more data to be kept server side. When the queue is full the server will disconnect the client to avoid "exploding" (a.k.a OutOfMemoryError)
Another note: SFS 1.6.0 has introduced finer configuration of the client queues, upgrade is highly recommended (and free
)
If you can't do it right now, at least update to 1.5.9
If clients receive lots of messages per second they might not always be able to keep up.
If this happens the server keeps data in the queue and waits the client to be able to receive more data.
You can configure the queue to allow more data to be kept server side. When the queue is full the server will disconnect the client to avoid "exploding" (a.k.a OutOfMemoryError)
Another note: SFS 1.6.0 has introduced finer configuration of the client queues, upgrade is highly recommended (and free
)If you can't do it right now, at least update to 1.5.9
Hi Lapo,
Thanks for your reply. It could be poor coding or poor planning on my side. But it would be really great if I can somehow have a way to find out if my rate of messaging is stressing the server. Is there any way to do this or it's just monitoring? I am not able to decide what's too much.
It seems to me sending a chunk of data once is better than sending series of small parts. Anything to higlight?
Regards,
darnpunk
Thanks for your reply. It could be poor coding or poor planning on my side. But it would be really great if I can somehow have a way to find out if my rate of messaging is stressing the server. Is there any way to do this or it's just monitoring? I am not able to decide what's too much.
It seems to me sending a chunk of data once is better than sending series of small parts. Anything to higlight?
Regards,
darnpunk
Hi Lapo,
I have updated the server to version 1.5.9 and there are improvements. There are lesser dropped messages but mass disconnection still occurs. I have set logging to FINER level and saw nothing that could lead to the disconnection. Is it possible to email the log file to your side for brief checks?
Regards,
darnpunk
I have updated the server to version 1.5.9 and there are improvements. There are lesser dropped messages but mass disconnection still occurs. I have set logging to FINER level and saw nothing that could lead to the disconnection. Is it possible to email the log file to your side for brief checks?
Regards,
darnpunk
From 1.5.9+ we have modified how we handle the message queues for each user.
1. monitor your bandwidth and make sure you are not running short of network resources, this could be a primary cause for this problem
2. disable the user count updates if you don't need them. They can consume significant network resources especially if you have hundreds of active rooms
3. adjust the settings in <ClientMessageQueue> of your config.xml
Check chapter 2.2 of the docs for all the details (Client message queue settings)
A few advices:<ClientMessagQueue>
<QueueSize>120</QueueSize>
<MaxAllowedDroppedPackets>1</MaxAllowedDroppedPackets>
</ClientMessagQueue>
1. monitor your bandwidth and make sure you are not running short of network resources, this could be a primary cause for this problem
2. disable the user count updates if you don't need them. They can consume significant network resources especially if you have hundreds of active rooms
3. adjust the settings in <ClientMessageQueue> of your config.xml
Check chapter 2.2 of the docs for all the details (Client message queue settings)
Hi Lapo,
Thanks for your advice. Message queues settings on the server side are set as follows:
1. Our allowed bandwidth is 10mbps and on average 6mbps is used up. I will continue to monitor this.
2. This has been disabled - uCountUpdate="false"
3. As above.
The dropped messages are lesser but disconnections still happens except lesser. From the logs, everything seems to be running fine prior to the mass disconnections.
I tried pinging the server (using codes from documentation) on peak hours and the calculated value is average at 11ms. It seems as if something is periodically disconnecting users for some unknown reason.
I do have multiples and continuous rooms being destroyed and created but I don't think those are "heavy"?
Regards,
darnpunk
Thanks for your advice. Message queues settings on the server side are set as follows:
Code: Select all
<ClientMessagQueue>
<QueueSize>120</QueueSize>
<MaxAllowedDroppedPackets>-1</MaxAllowedDroppedPackets>
</ClientMessagQueue> 2. This has been disabled - uCountUpdate="false"
3. As above.
The dropped messages are lesser but disconnections still happens except lesser. From the logs, everything seems to be running fine prior to the mass disconnections.
I tried pinging the server (using codes from documentation) on peak hours and the calculated value is average at 11ms. It seems as if something is periodically disconnecting users for some unknown reason.
I do have multiples and continuous rooms being destroyed and created but I don't think those are "heavy"?
Regards,
darnpunk
