sendResponse being received out of order or all at once

[Lapo]

However, the other client will sometimes receive a cluster of messages in one big lump.

This is quite possible. Very small messages tend to be aggregated by the TCP protocol. There is no control over this aspect and it shouldn't affect your application in 99.9% of the cases.

Client A will receive these messages in the correct order and with up to a second in between, or almost instantly if the server logic is very light.
The other client will send its action message and then trace nothing for upwards of 30 seconds.

What happens then? Do these messages arrive?
Can you reproduce this problem locally? Because over the internet the 2nd client might just be "stuck". If this is the case it will never receive anything and will be eventually disconnected.

This IS a socket connection and we have verified that the client just previously sent a message and received a response within less than a second. Yet, in this case the next several messages would come in one bunch.

Clear. But what's the problem with the bunch, exactly?

As gentleman has pointed out, quite often the client won't receive a message until after it has sent a message of its own.

Can you reproduce this in a local environment?
If so can you provide us a way to reproduce it?

Client A finishes animating and sends the message to the server, Client B however doesn't trace out the update from the server in the first place.

In other words you are saying that:
- Client A sends request to Server
- Server generates messages for Client A and B
- Client A gets the response immediately, but client B doesn't

Correct?
Are you sure your Client B isn't too busy (Flash Player CPU usage) working on other things that it doesn't have any more time to handle any network events?
If you are running a tight, heavy animation loop this can happen.

[VengantMjolnir]

However, the other client will sometimes receive a cluster of messages in one big lump.

This is quite possible. Very small messages tend to be aggregated by the TCP protocol. There is no control over this aspect and it shouldn't affect your application in 99.9% of the cases.

What would you consider a small message? We are using json protocol and the messages can exceed a thousand characters as a raw string.

Client A will receive these messages in the correct order and with up to a second in between, or almost instantly if the server logic is very light.
The other client will send its action message and then trace nothing for upwards of 30 seconds.

What happens then? Do these messages arrive?

No, the messages do not arrive. Or at least, the socket never triggers the onProgress event. I know this for sure because of the minor modification I made to SmartFox's API. I added a custom traceOut call to the debugmessage method inside SmartFoxClient.as. So all received messages are dumped before my client even processes them at the application level.

Can you reproduce this problem locally? Because over the internet the 2nd client might just be "stuck". If this is the case it will never receive anything and will be eventually disconnected.

We have never been able to reproduce the problem locally. I am quite sure that the other client is not 'stuck', allthough my definition of stuck would be max CPU/frozen. This is not the case since their timer events are still firing and they eventually send a timeout message.

This IS a socket connection ... the next several messages would come in one bunch.

Clear. But what's the problem with the bunch, exactly?

The problem is that the client is in one phase state, waiting for server instruction to proceed. It is a dumb-client that acts as the view for the server logic and as such only contains enough logic to display choices to the player and forward those to the server. The messages from the server are supposed to keep both clients in sync( or within the buffer tolerance of sync ) to each other.

As gentleman has pointed out, quite often the client won't receive a message until after it has sent a message of its own.

Can you reproduce this in a local environment?
If so can you provide us a way to reproduce it?

I have not been able to reproduce this in a local environment. Have you never heard of anything similar?

Client A finishes animating and sends the message to the server, Client B however doesn't trace out the update from the server in the first place.

In other words you are saying that:
- Client A sends request to Server
- Server generates messages for Client A and B
- Client A gets the response immediately, but client B doesn't

Correct? Are you sure your Client B isn't too busy...?

Correct.
I am quite sure that the client is not locked up in an infinte loop. Because of the 'elastic racetrack' of flash's frame model I can be sure that my code is being run within the time-lapse we are seeing. IF it did take longer than 15 seconds, I would see an error 1502 from a script timeout. If not, then enter frame updates would be running. Plus, like I said earlier, the client has a timer running that eventually times out.

Here is a timeline of a sample phase
1:00 - Server sends beginPhase
1:00 - Client A and B receive beginPhase Client B is active
1:05 - Client B sends action, waits for other player
1:05 - Server sends updatePhase
1:05 - Client B receives updatePhase and resets timer with 45+15 second buffer
2:00 - Client A sends timeout to server
2:00 - Client A receives updatePhase
2:05 - Client B sends timeout to server

Client A sends the timeout 60 seconds( 45 + 15 ) after it received the beginPhase message because it didn't receive the update phase message.

[Lapo]

I have not been able to reproduce this in a local environment. Have you never heard of anything similar?

Never.

In other words you are saying that:
- Client A sends request to Server
- Server generates messages for Client A and B
- Client A gets the response immediately, but client B doesn't

Correct? Are you sure your Client B isn't too busy...?

Correct.

This is where I am lost, because this is so basic that it would take nothing to reproduce it. If you are not able to do so then we're talking about, maybe, an erratic problem due to a sloppy client connection.

I am quite sure that the client is not locked up in an infinte loop. Because of the 'elastic racetrack' of flash's frame model I can be sure that my code is being run within the time-lapse we are seeing.

I agree but this would anyways explain a few things.
The "bursts" of messages for examples. If the player is under heavy pressure because of complex rendering it will not process anything else until it passes that critical phase (which may last just a few seconds)
Data from sockets will continue to pile up and on the next rack-track loop all message will appear as a bunch at the same time.
Of course is pretty simple to spot this case, you just need to monitor the cpu consumption of the Flash Player on the client machine.

Can you better explain the phase section, I don't know what the game does and it's not clear what happens.

1:05 - Client B receives updatePhase and resets timer with 45+15 second buffer

what is a 45+15 buffer?

2:00 - Client A sends timeout to server

What is this? Can you explain?

[Lapo]

No, the messages do not arrive. Or at least, the socket never triggers the onProgress event. I know this for sure because of the minor modification I made to SmartFox's API.

BTW, which onProgress event are you referring to?
The API only listen for actual data coming from the socket via the ProgressEvent.SOCKET_DATA event. Is this the event that you are monitoring in your custom traces?

[VengantMjolnir]

the progress event I was reffering to is the ProgressEvent.SOCKET_DATA event that triggers SmartFoxClient::handleSocketData.

You'r right, it should take nothing to reproduce. Just play the game normally under different client loads to see if it comes up. I even implimented a frame rate limiter( timed while loop ) that jacks the cpu up to 100%. I've tested this on dual cores, single cores and the result is the same.. no error. I don't have access to any of the client's machines so I built the traceOut function to store debug messages from smartfox and the game. At the end of the game, the log is sent to a webserver and stored along with the server log for that game.

I could understand a burst of messages being as a result of a few seconds of animation delay, but not upwards of 30.

I can explain the 'phase section' in greater detail. It is a multi player collectible card game ( http://leagueofsuperevil.com ). The game is divided into seperate phases, each having an enter and exit method. During the game there are a few of the phases were back and forth action is available. Those phases have startphase and endphase methods as well. (I'll call the back and forth phases co-op phases for clarity)

During a non co-op phase, each client will receive a begin phase message. This causes the state machine to call exit on prev phase, and enter on the new one. Enter sets up the timer for the phase and adds network listeners for messages its interested in. The active player has a timer equal to the phase-time, while the inactive player has a 15 second buffer( that's what 45+15 was. ) The server sends a list of playable cards to the active player and they either make a selection, pass( via a button press ) or timeout. Timeouts from active players triggers a pass.
Server then processes the response and sends back any animations that need to be played. Each client sends an animation done message( to keep them synced and the server sends a begin phase for the next phase.

During a co-op phase, it is largely the same at first. However after a client has performed an action the server sends an update phase and active players switch. Once two passes are received, the server processes the responses and sends back animations. Resolution is the sames as above.

The problem we are describing could come at ANY point in there... it could be a client sends an animation done and waits on the response( with a time ticking down. ) Eventually, it times out and sends a message to the server, then it receives the update phase and any other messages it missed. Or it could be in another part.

I'll attempt to explain the timeout a bit more. It is a timer on the client that counts down from a specific time to zero, represented visually by a radial wipe. In order to limit the amount of processing on the server, we implemented a timer that is on each of the clients. When phases begin and update, and during animation, the timer is set. When it runs to zero, an event is triggered and processed based on wether they are the active client or not. By giving a buffer, we are supposed to be able to guarantee that an active client will always time out first, thereby passing. If the inactive client times out, the game ends as a timeout. In theory and in testing this worked flawlessly. However in our open beta we have run into a number of games that failed.

If this is due to erratic internet connection.. does smartfox ping the client? What is the purpose of

Code: Select all

if (msg != "ok")

that is in handleMessage(). [/code]

[Lapo]

I could understand a burst of messages being as a result of a few seconds of animation delay, but not upwards of 30.

Yes, me too

Each client sends an animation done message( to keep them synced and the server sends a begin phase for the next phase.

Hmmm, this is potentially dangerous.
The client might have any problem and never return the "done" message or delay it by a pretty long time.
It is advisable to use time-based animations so that you always know how much time an animation will take on the client, regardless of its rendering performance. I know, it's harder to do but it never loose synchronization.

In order to limit the amount of processing on the server, we implemented a timer that is on each of the clients.

Allow me to say that this assumption is not correct. The server can handle thousands of timed events without a problem. Also this would add better security and synchronization.
We provide different mechanisms to achieve this using the Scheduler objects. In particular we highly recommend the ScheduledThreadPoolExecutor object provided by Java 5 and higher.
More details are found in the Java Cookbook here:
http://www.smartfoxserver.com/docs/docP ... okBook.htm

[pandiri]

HI Lapo,

Any update on this issue, looks like we are facing a similar issue !

Thanks in Advance..
Ravinder

[Lapo]

Hi,
did you read the whole discussion? You should find there several things to check on your side.
Then please report all of the details about the problem, because although you might get the same problems the causes are very different from one case to the other.

Please let us know the details.