Bottleneck on sending UDP packets

[JohnHacker]

Hello,

After lot of research we just found we were bottlenecked because we were using UDP packets to thousands of users and SFS is using just one DatagramChannel to send all of them, so even using lots of threads for the SocketWriter, they are all blocked waiting for the synchronized inside DatagramChannelImpl. Why isn't the UDP system using a pool of channels? It's not scalling at all and we are having a lot of trouble. As for now, we used a workaround by sending all that information through TCP/IP, but it's information that we don't need to be reliable and it would be the best to use UDP for it.

Thank you!

[Lapo]

Hi,
what SFS2X server version are you running?
What client side API and platform do you use?

Also, what is the average packet rate of your UDP requests?
How many players per Room?

Thanks

[JohnHacker]

We use the last version, 2.14. All the other questions have nothing to do with the problem.

[Lapo]

They have to do with it because we need to test it.
Can you please answer the questions?

Thanks

[JohnHacker]

It's over 30k users, each receiving 12-18 UDP packets per second. All the UDP packets are being sent through the same socket and there's a semaphore on the Java implementation of the UDP socket (DatagramChannelImpl), so it can't send more than 60-80 kpps, no matter the number of cores you use.

[Lapo]

Thanks.
We're working on an update patch which will also require at least a couple of days of stress testing.
We'll post an update next week.

Cheers

[JohnHacker]

Thanks!

[Lapo]

A quick update on the topic.
We've been researching and experimenting with the DatagramChannel issue and have noticed that, generally speaking, this problem seems mostly unknown. What I mean by this is that, even though the Javadoc does include a comment relative to the single threadedness of the DatagramChannel class, there are no indications or best practices to scale it on the server side.

Also, there're almost no resources/discussions in technical forums, nor we have found any analysis of this issue in books dedicated to the topic of Java networking.

I did find a very similar discussion in the Netty repository but no solutions were proposed, and the bug report is still open after 8 years.

We're now testing several possible solutions using both non-blocking and blocking UDP sockets, but we need time to compare the behavior of these solutions under different loads and also under different operating systems, to make sure everything works as expected.

For the time being I don't have a timeline for the patch release. We're now moving this issue at the top of our priority list.
Hopefully we should be able to release an update within 2-3 weeks, but don't quote me on that

I'll post more updates as soon as they are available.

[Lapo]

UPDATE:
We have finished a series of larger stress test on the update we're preparing for an official release during the next 7-10 days.
To give you an idea, we have been testing on a quad-core Xeon E3-1578L w/ 32GB RAM, running 1700 CCU.
The users are organized in game Rooms of 8 players, and each player is sending 30 updates per second using UDP.
This in turn generates 8x30 = 240 req/s per Room. There are 212 Rooms so => 50880 req/sec

The server then replies broadcasting each update to all players in the Room, which gives:
(8x30) x 8 players = 1920 packets/Room
1920pps x 212 Rooms = 407K+ pps (outgoing)

We ran the test on both SFS2X 2.14.0 and the new patch 2.14.1.

SFS2X 2.14.0:
Struggles keeping up with the requests outputting a max of 28MB/s and 215Kpps
Outoging message queue is always full
Thread contention is high, especially in the SocketWriter department (see attached pic)

vvm_2_14_0.jpg

(176.07 KiB) Not downloaded yet

SFS2X 2.14.1:
Keeps up with the test, outputting 52MB/s and 400K+ pps
Outgoing message queue is moderately busy
There're no signs of thread contention as per the attached screenshot

vvm_2_14_1.jpg

(170.61 KiB) Not downloaded yet

@JohnHacker: I would suggest to get in touch with us next week via our support@... email box. We should be able to send you a pre-release of the patch to test on your side.

Hope it helps

[JohnHacker]

Hi! I sent an e-mail