Too much Dropped message after 500 connections

Kleon · Post by **Kleon** » 29 Sep 2010, 12:07

Hello,

We have a simple chat roulette application (just text chat) with typing notification and carma tool. But when the number of connection is beign more than 500, then too much Dropped messages are appeared. At the same time the admin tool shows 0% of server load and a server monitor shows 1%.

We checked bandwidth on the server side and it's ok.

Could you please provide us any suggestion how we can resolve this problem?

P.S. SmartfoxServerPRO was installed to our server with standart properties.

Kleon · Post by **Kleon** » 29 Sep 2010, 13:12

Our server bandwidth is 100Mbs.
Here is our Data traffic statistic within half an hour

Data IN: 8Mb
Data OUT: 538Mb

BigFIsh · Post by **BigFIsh** » 29 Sep 2010, 19:20

Is this a local testing environment? i.e. are you making those connections?

Anyhow, 'dropped messages' have been discussed many times in this forum, more than I can count. Please use the search tool.

Kleon · Post by **Kleon** » 30 Sep 2010, 06:07

BigFIsh wrote:Is this a local testing environment? i.e. are you making those connections?

No, it's a live server with real connections.

Yes, I have investigated these topics. As far I understood, that here is 3 major reasons why dropped messages may appear:
1. Bandwidth problem
2. Memory problem
3. Poor server capacity

But, it seems we haven't anyone from it.

Here is statistic from our hosting service:
We are barely using any bandwitdh:
http://img.skitch.com/20100929-j3gq732c ... 8gkwq6.jpg

We are barely using any of our 2G of memory:
http://img.skitch.com/20100929-e5fnpx21 ... 14fmdu.jpg

Also Smartfox admin tool and server monitor tool shows 1-4% of server loading.

So we are little confused what is going on.
Could JVM be as bottleneck? We just installed Smartfox on standard hosting solution and added "-server" to SFS start script.

Could you suggest us next steps to find a problem?

BigFIsh · Post by **BigFIsh** » 30 Sep 2010, 07:21

Yes, I have investigated these topics. As far I understood, that here is 3 major reasons why dropped messages may appear:
1. Bandwidth problem
2. Memory problem
3. Poor server capacity

Actually, I think you may have missed some main reasons for dropped messages. Have you read and understood the "Understanding Dropped Messages" section of this documentation?

Furthermore, what are your dropped messages value within half a hour?

Kleon · Post by **Kleon** » 30 Sep 2010, 07:59

Actually, I think you may have missed some main reasons for dropped messages. Have you read and understood the "Understanding Dropped Messages" section of this documentation?

Furthermore, what are your dropped messages value within half a hour?

Do you mean "Bad or slow client connection" reason?

Dropped message value - 2-3k. It starts to increase immediately after 450-550 connections. Until the number of connection is less, dropped message = 0. After "critical point", the server disconnects 100-200 users and the system is stabilized.

BigFIsh · Post by **BigFIsh** » 30 Sep 2010, 09:00

Yes, that's correct. Clients (that has slow internet connection) are getting dropped due to the fact that they can't keep up with the incoming messages from the server. The max allowed drop messages is defined in the config.xml file. A tight max allowed drop message value keeps all clients in sync with each other.

You will need to decrease the number of clients per room (up to 30 per room is recommended) or decrease the rate and/or intensity of data being sent from client and server.

Kleon · Post by **Kleon** » 30 Sep 2010, 10:26

Yes, that's correct. Clients (that has slow internet connection) are getting dropped due to the fact that they can't keep up with the incoming messages from the server. The max allowed drop messages is defined in the config.xml file. A tight max allowed drop message value keeps all clients in sync with each other.

Thanks for clarification. In our situation server haven't dropped messages or have it little (1-10) until the amount of connections is less than 450-550. After this "critical point", clients that have been worked stable are dropped too. It looks like if we have some "bottleneck problem" on the server.

Average data exchanging for one client is 20-40 messages (client<->server) per minute.
If we launch 2 clients within local environment, then start one minute application chat session. Smartfox admin tool shows following Data traffic statistic (per minute):
Data IN: 7.3 KB
Data OUT: 45.6 KB

Also we have only one "Lobby" room that can contain more than 2 users. Here is Lobby properties:

Code: Select all

			<Rooms>
				<Room name="Lobby" isPrivate="false" isTemp="false" limbo="true" autoJoin="true" />
			</Rooms>

And here is the object which apply to all dynamically created rooms:

Code: Select all

			var roomObj = {};
			roomObj.name = roomName;
			roomObj.maxU = 2;
			//
			roomObj.sendUpdate = false;
			roomObj.uCount = false;

In the peak load (when dropped messages appear) we use only 6-8Mbps from 100Mbps of our server bandwidth.

Does it confirm that data rate and intensity is ok? If so, what else can we test?

BigFIsh · Post by **BigFIsh** » 30 Sep 2010, 19:47

One more question that I forgot to ask - in your AdminTool.swf, was it the dropped incoming messages or dropped outgoing messages in which the values are high? Additionally, did you get any warnings in your log file when the disconnection had occurred?

Lastly, your SFS & Client API version?

Kleon · Post by **Kleon** » 01 Oct 2010, 07:10

BigFIsh wrote:One more question that I forgot to ask - in your AdminTool.swf, was it the dropped incoming messages or dropped outgoing messages in which the values are high?

It was outgoing dropped messages - 2-3k. The value of incoming dropped messages is 0.

Additionally, did you get any warnings in your log file when the disconnection had occurred?

No, Application logic works successfully correctly without errors in Logs.

Lastly, your SFS & Client API version?

SmartFoxServer PRO - 1.6.8
SmartFoxClient - 1.6.1

Post by **Lapo** » 01 Oct 2010, 08:47

Hi,
BigFish mentioned the max number of Users per Room.
Sounds like this could be the problem. How do you distribute users in Rooms? Do they 500 users all access the same Room?

Kleon · Post by **Kleon** » 01 Oct 2010, 13:34

Lapo wrote:Hi,
BigFish mentioned the max number of Users per Room.
Sounds like this could be the problem. How do you distribute users in Rooms? Do they 500 users all access the same Room?

Yes. All users connect to the Lobby room first.

Code: Select all

        
<Rooms> 
            <Room name="Lobby" isPrivate="false" isTemp="false" limbo="true" autoJoin="true" /> 
         </Rooms>

Then server logic finds a pair of users and creates room for 2 users dynamically.

Code: Select all

         var roomObj = {}; 
         roomObj.name = roomName; 
         roomObj.maxU = 2; 
         // 
         roomObj.sendUpdate = false; 
         roomObj.uCount = false;

So for 500 users we have one Lobby room which contains 500 users and about 250 dynamically created rooms which contain pair of users

BigFIsh · Post by **BigFIsh** » 01 Oct 2010, 18:35

This 'Lobby' room, what does it do? Does it list down all game rooms? Does it dynamically update room variables? Sending information (such as room variables, user count, and so on) for 250 rooms to each user in the lobby can stack up quite quickly.

Post by **Lapo** » 02 Oct 2010, 05:27

I would like to see a network monitor graph showing the progression of the traffic as you reach those 500 users.
Do you monitor your bandwidth?

Kleon · Post by **Kleon** » 04 Oct 2010, 05:27

This 'Lobby' room, what does it do? Does it list down all game rooms? Does it dynamically update room variables? Sending information (such as room variables, user count, and so on) for 250 rooms to each user in the lobby can stack up quite quickly.

It's a one room which client connects first. For the Lobby room I set property limbo = true. As far I understood from the documentation this property means that the room doesn't interact with users.

Also I set property uCountUpdate="false" on the Zone level.

I would like to see a network monitor graph showing the progression of the traffic as you reach those 500 users.
Do you monitor your bandwidth?

Yes. Here is it. We have about 500 connections on Wednesday.
http://img.skitch.com/20100929-j3gq732c ... 8gkwq6.jpg