Issue: Too many dropped messages

Praveen · Post by **Praveen** » 27 Jun 2009, 11:07

Hi,

I am trying to load test SmartFoxServer, over the LAN - latency <1ms, for which I have created an Executable that creates 50 users/connections to SFS. From each user/connection I am sending 10 position updates per second to SFS, using SetUserVariables API method.

After a few seconds of execution SFS is disconnecting the clients stating that 'Too many dropped messages' and I can see the 'Dropped Out-Msg' counter on the SFS Admin page incremented.

I have been through the User documentation on SFS website
http://www.smartfoxserver.com/docs/docP ... ooting.htm and it says that 'it is not recommended to to send more than 10-20 updates per second, unless you are in a Local Network'. Since I am on the LAN why am I facing this problem, is there any restriction of messages being sent from Client-Server and vice-versa?[/url]

Post by **Lapo** » 29 Jun 2009, 06:32

Hi,
are you using a cable connection or wireless?
Sounds like it's wireless...

Praveen · Post by **Praveen** » 29 Jun 2009, 07:53

Its Wired.

Post by **Lapo** » 29 Jun 2009, 08:10

I think the problem is caused by the clients which are not able to keep up with the amount of data they receive. Therefore their network queue gets very busy and the server cannot send more data to them.

50 users sending 10 msg/sec == 500 msg/sec
If these 50 users are inside the same room it means that every second the server is broadcasting 500 x 50 = 25.000 messages back.

I don't know how you run these tests ( one single machine? How many Flash Player instances? ) but it seems alot like the client side is not keeping up, hence the dropped messages problem.

Praveen · Post by **Praveen** » 29 Jun 2009, 09:03

SFS Server and the Client system (on which I have my application running) are both hosted on VM Ware but on Quad Core processor systems.

Let me know your comments if this would impact in any way?

Post by **Lapo** » 29 Jun 2009, 10:59

I think the problem could be related with the massive amount of messages that your test client has to deal with.
In a real case scenario those 50 users would each run on their machines receiving only their data.

Also is this scenario realistic? Is your game going to handle 50 players in the same room playing with each others in realtime? That might be a little risky for Flash. While I am sure most Flash-based games can handle 15-25 realtime users, I think that going up to 50 would exclude a high percentage of possible players.

I suspect that 50 users in the same room require a lot of rendering (moving all the players and I presume also scrolling) and quite a bit of work in handling the network traffic. Slower computers and connections will not be able to cope with this. (hope to be wrong)

Praveen · Post by **Praveen** » 29 Jun 2009, 11:45

I could get over the massive incoming messages by creating an Extension and in their have filtered out some msgs. Now everything is going fine.

But had a new problem when my client load has increased to 500; initially SFS was dropping Incoming Messages and later Outgoing messages were dropped and eventually disconnected the clients.

In the config file 'MaxIncomingQueue' value is '5000', does it help increasing this value to '10000'? In my case I have 500 users i.e. they can send 5000 msgs/sec.

Post by **Lapo** » 29 Jun 2009, 11:50

But had a new problem when my client load has increased to 500;

500 in the same room?

Praveen · Post by **Praveen** » 29 Jun 2009, 12:33

Yes! This is a test application Lapo.

Would the behavior of SFS wary if we have 500 concurrent users logged in to 20 Rooms (each room containing 25 users); and all of them sending 10 msgs/sec?

Post by **Lapo** » 29 Jun 2009, 13:09

Yes, it makes a substantial difference!

500 users in one single room means that one single message has to be broadcasted back to the other 500 (or 499) users. Very large broadcasts like these are definitely heavier than frequent smaller broadcasts.

So if you need to do a realistic test you should split the players in rooms

Praveen · Post by **Praveen** » 29 Jun 2009, 13:22

Yes lapo.

I am gonna doit tomorrow and will get let you know the results.

In the meantime can you tell me the use of 'MaxIncomingQueue' config element in the configuration file, I did not find any documentation on this.

Post by **Lapo** » 29 Jun 2009, 13:33

That param rarely needs to be changed. It specifies the max number of messages that can stay in the incoming process queue.
You can adjust it if you find the server dropping incoming messages.

Praveen · Post by **Praveen** » 01 Jul 2009, 16:23

Hi,

I am testing now with 100 users in 4 rooms i.e. 25 users in 1 room, but I have the same problem of Too Many Dropped Out-messages.

Is there anyway that we can configure SFS to send outbound messages on a different port number than 9339?

Post by **Lapo** » 02 Jul 2009, 05:09

I am testing now with 100 users in 4 rooms i.e. 25 users in 1 room, but I have the same problem of Too Many Dropped Out-messages.

1. It is very likely that the client application is not able to keep up wih the mass of messages received
2. You can configure the server to be more tolerant with slow clients and keep more data inside each User queue before dropping.
See chapter 2.1 of the docs.
3. Last but not least: check your bandwidth. If it is almost full messages will travel very slowly filling up the server queue

Is there anyway that we can configure SFS to send outbound messages on a different port number than 9339?

Incoming and outgoing traffic travel on the same port, otherwise you would need two connections. You can change the port used by SFS by altering it in the config.xml

Praveen · Post by **Praveen** » 02 Jul 2009, 06:09

I have installed SFS Pro trial version (allows 20 users); on my system and the Client is also running on my system.

I am testing with 19 users sending 10msgs per second; even then after a few seconds SFS disconnects all the users saying 'Too Many Dropped Messages'.

There is no latency, nothing at all. I have modified the ClientMessagQueue in the config.xml as follows:
<ClientMessagQueue>
<QueueSize>120</QueueSize>
<MaxAllowedDroppedPackets>1</MaxAllowedDroppedPackets>
</ClientMessagQueue>

FYI: I am using .Net Client API. You may ask me to increase MaxAllowedDroppedPackets to higher value, but I dont expect the clients to loose position updates.

Any thoughts on this.