Server OutOfMemoryError

Mather · Post by **Mather** » 17 Jun 2010, 15:54

Hello,

We've been having some problems with the project we are running. The server is constantly running however it crashes at least once a day giving errors that Java has run out of virtual memory. The number of clients connected at the time doesnt seem to affect the crash and it can crash between 5 and 30 hours after being restarted. We increased the amount of memory that smartfox is allocated on startup using the -Xms64M -Xmx1500M approach, but this has not helped.

The smartfox log states this when it crashes:

Code: Select all

#
# An unexpected error has been detected by Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 1143304 bytes for Chunk::new. Out of swap space?
#
#  Internal Error (allocation.cpp:218), pid=4924, tid=1296
#  Error: Chunk::new
#
# Java VM: Java HotSpot(TM) Server VM (11.3-b02 mixed mode windows-x86)
# An error report file with more information is saved as:
# C:\Program Files (x86)\SmartFoxServerPRO_1.6.6\Server\hs_err_pid4924.log
#
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

And the error log that this creates is also shown below:

Code: Select all

#
# An unexpected error has been detected by Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 1143304 bytes for Chunk::new. Out of swap space?
#
#  Internal Error (allocation.cpp:218), pid=4924, tid=1296
#  Error: Chunk::new
#
# Java VM: Java HotSpot(TM) Server VM (11.3-b02 mixed mode windows-x86)
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

---------------  T H R E A D  ---------------

Current thread (0x67106400):  JavaThread "CompilerThread0" daemon [_thread_in_native, id=1296, stack(0x675a0000,0x675f0000)]

Stack: [0x675a0000,0x675f0000]
[error occurred during error reporting (printing stack bounds), id 0xc0000005]


Current CompileTask:
C2:1261  !   it.gotoandplay.smartfoxserver.controllers.SystemHandler.processEvent(Lit/gotoandplay/smartfoxserver/lib/ServerEvent;)V (1208 bytes)


---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )
  0x705dec00 JavaThread "__evictor__" [_thread_blocked, id=3620, stack(0x6d350000,0x6d3a0000)]
  0x679bec00 JavaThread "pool-14-thread-1" [_thread_blocked, id=3232, stack(0x6b200000,0x6b250000)]
  0x7164a800 JavaThread "Timer-12" [_thread_blocked, id=4956, stack(0x6b120000,0x6b170000)]
  0x6886d800 JavaThread "selector" [_thread_blocked, id=4172, stack(0x6af20000,0x6af70000)]
  0x68e62c00 JavaThread "pool-13-thread-1" [_thread_blocked, id=1860, stack(0x6ad30000,0x6ad80000)]
  0x68e62800 JavaThread "pool-12-thread-1" [_thread_blocked, id=2744, stack(0x6aca0000,0x6acf0000)]
  0x6bfc5400 JavaThread "pool-11-thread-1" [_thread_blocked, id=2344, stack(0x6ac10000,0x6ac60000)]
  0x6b6be400 JavaThread "pool-10-thread-1" [_thread_blocked, id=1092, stack(0x6ab80000,0x6abd0000)]
  0x67ef7c00 JavaThread "pool-9-thread-1" [_thread_blocked, id=3612, stack(0x6aaf0000,0x6ab40000)]
  0x6b3f2400 JavaThread "xDispatcher#CLIENT" daemon [_thread_in_native, id=3992, stack(0x6aa60000,0x6aab0000)]
  0x6849b800 JavaThread "xIoTimer" daemon [_thread_blocked, id=4252, stack(0x6a8f0000,0x6a940000)]
  0x678da400 JavaThread "Timer-11" [_thread_blocked, id=4784, stack(0x6a480000,0x6a4d0000)]
  0x67ceb800 JavaThread "ExtensionHandler-1" daemon [_thread_blocked, id=3028, stack(0x6a9d0000,0x6aa20000)]
  0x6799b400 JavaThread "BlueBoxHandler-1" daemon [_thread_blocked, id=3484, stack(0x6a980000,0x6a9d0000)]
  0x67f1d800 JavaThread "SystemHandler-1" daemon [_thread_blocked, id=1536, stack(0x6a5f0000,0x6a640000)]
  0x67f1d400 JavaThread "EventWriter-2" daemon [_thread_blocked, id=728, stack(0x6a560000,0x6a5b0000)]
  0x678dac00 JavaThread "EventWriter-1" daemon [_thread_blocked, id=2100, stack(0x6a4d0000,0x6a520000)]
  0x690cb400 JavaThread "H2 TCP Server (tcp://localhost:9009)" [_thread_in_native, id=4260, stack(0x6a3f0000,0x6a440000)]
  0x67971c00 JavaThread "Timer-10" daemon [_thread_blocked, id=4720, stack(0x69890000,0x698e0000)]
  0x6f513800 JavaThread "Timer-9" daemon [_thread_blocked, id=1356, stack(0x687d0000,0x68820000)]
  0x6b6bec00 JavaThread "Timer-8" daemon [_thread_blocked, id=392, stack(0x68740000,0x68790000)]
  0x68968800 JavaThread "Timer-7" daemon [_thread_blocked, id=4808, stack(0x6a2d0000,0x6a320000)]
  0x67b59c00 JavaThread "btpool1-9 - Acceptor0 SelectChannelConnector@0.0.0.0:8080" [_thread_in_native, id=2212, stack(0x6a110000,0x6a160000)]
  0x680f8800 JavaThread "btpool1-8 - Acceptor1 SelectChannelConnector@0.0.0.0:8080" [_thread_in_native, id=4212, stack(0x6a080000,0x6a0d0000)]
  0x68def400 JavaThread "btpool1-7" [_thread_blocked, id=3288, stack(0x69ff0000,0x6a040000)]
  0x68a88c00 JavaThread "btpool1-6" [_thread_blocked, id=4960, stack(0x69f60000,0x69fb0000)]
  0x68519800 JavaThread "btpool1-5" [_thread_blocked, id=5052, stack(0x69c50000,0x69ca0000)]
  0x680a8c00 JavaThread "btpool1-4" [_thread_blocked, id=4380, stack(0x69bc0000,0x69c10000)]
  0x68ae6000 JavaThread "btpool1-3" [_thread_blocked, id=3796, stack(0x69b30000,0x69b80000)]
  0x67cfe800 JavaThread "btpool1-2" [_thread_blocked, id=3812, stack(0x69aa0000,0x69af0000)]
  0x68a89400 JavaThread "btpool1-1" [_thread_blocked, id=3644, stack(0x69a10000,0x69a60000)]
  0x695aa800 JavaThread "btpool1-0" [_thread_blocked, id=3588, stack(0x69980000,0x699d0000)]
  0x68ea6c00 JavaThread "SmartFoxServer" [_thread_in_native, id=1012, stack(0x68600000,0x68650000)]
  0x67b7bc00 JavaThread "__evictor__" [_thread_blocked, id=4116, stack(0x6afb0000,0x6b000000)]
  0x68ad4400 JavaThread "xDispatcher#CLIENT" daemon [_thread_in_native, id=4964, stack(0x6ae90000,0x6aee0000)]
  0x68076400 JavaThread "xIoTimer" daemon [_thread_blocked, id=1940, stack(0x6ae00000,0x6ae50000)]
  0x680af000 JavaThread "MySQL Statement Cancellation Timer" daemon [_thread_blocked, id=4120, stack(0x6a860000,0x6a8b0000)]
  0x68c4a400 JavaThread "Thread-23" [_thread_blocked, id=4664, stack(0x6a7d0000,0x6a820000)]
  0x68c30400 JavaThread "Timer-4" daemon [_thread_blocked, id=4104, stack(0x6a1a0000,0x6a1f0000)]
  0x00447000 JavaThread "DestroyJavaVM" [_thread_blocked, id=1368, stack(0x00510000,0x00560000)]
  0x680be400 JavaThread "PooledByteBufferExpirer-0" daemon [_thread_blocked, id=1768, stack(0x686b0000,0x68700000)]
  0x6710e800 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=4528, stack(0x676c0000,0x67710000)]
  0x6710b000 JavaThread "CompilerThread1" daemon [_thread_blocked, id=4636, stack(0x67630000,0x67680000)]
=>0x67106400 JavaThread "CompilerThread0" daemon [_thread_in_native, id=1296, stack(0x675a0000,0x675f0000)]
  0x67104c00 JavaThread "Attach Listener" daemon [_thread_blocked, id=5048, stack(0x67510000,0x67560000)]
  0x67103800 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=1656, stack(0x67480000,0x674d0000)]
  0x670f0400 JavaThread "Finalizer" daemon [_thread_blocked, id=3408, stack(0x673f0000,0x67440000)]
  0x670ef000 JavaThread "Reference Handler" daemon [_thread_blocked, id=3948, stack(0x67360000,0x673b0000)]

Other Threads:
  0x670eb800 VMThread [stack: 0x672d0000,0x67320000] [id=3056]
  0x67110400 WatcherThread [stack: 0x67750000,0x677a0000] [id=2544]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap
 PSYoungGen      total 20160K, used 7953K [0x5c320000, 0x5dcc0000, 0x669c0000)
  eden space 17024K, 28% used [0x5c320000,0x5c7dbac0,0x5d3c0000)
  from space 3136K, 99% used [0x5d460000,0x5d7689d0,0x5d770000)
  to   space 4608K, 0% used [0x5d840000,0x5d840000,0x5dcc0000)
 PSOldGen        total 58304K, used 34742K [0x08dc0000, 0x0c6b0000, 0x5c320000)
  object space 58304K, 59% used [0x08dc0000,0x0afada00,0x0c6b0000)
 PSPermGen       total 27008K, used 26644K [0x04dc0000, 0x06820000, 0x08dc0000)
  object space 27008K, 98% used [0x04dc0000,0x067c5048,0x06820000)

Dynamic libraries:
0x00400000 - 0x00424000 	C:\Program Files (x86)\SmartFoxServerPRO_1.6.6\jre\bin\java.exe
0x7d600000 - 0x7d6f0000 	C:\WINDOWS\system32\ntdll.dll
0x7d4c0000 - 0x7d5f0000 	C:\WINDOWS\syswow64\kernel32.dll
0x00360000 - 0x003fb000 	C:\WINDOWS\syswow64\ADVAPI32.dll
0x7da20000 - 0x7db00000 	C:\WINDOWS\syswow64\RPCRT4.dll
0x7d8d0000 - 0x7d920000 	C:\WINDOWS\syswow64\Secur32.dll
0x7c340000 - 0x7c396000 	C:\Program Files (x86)\SmartFoxServerPRO_1.6.6\jre\bin\msvcr71.dll
0x6db30000 - 0x6def4000 	C:\Program Files (x86)\SmartFoxServerPRO_1.6.6\jre\bin\server\jvm.dll
0x7d930000 - 0x7da00000 	C:\WINDOWS\syswow64\USER32.dll
0x7d800000 - 0x7d890000 	C:\WINDOWS\syswow64\GDI32.dll
0x76aa0000 - 0x76acd000 	C:\WINDOWS\system32\WINMM.dll
0x71bc0000 - 0x71bc8000 	C:\WINDOWS\system32\rdpsnd.dll
0x771f0000 - 0x77201000 	C:\WINDOWS\system32\WINSTA.dll
0x77ba0000 - 0x77bfa000 	C:\WINDOWS\syswow64\msvcrt.dll
0x71c40000 - 0x71c97000 	C:\WINDOWS\syswow64\NETAPI32.dll
0x76b70000 - 0x76b7b000 	C:\WINDOWS\system32\PSAPI.DLL
0x6d340000 - 0x6d348000 	C:\Program Files (x86)\SmartFoxServerPRO_1.6.6\jre\bin\hpi.dll
0x6d860000 - 0x6d86c000 	C:\Program Files (x86)\SmartFoxServerPRO_1.6.6\jre\bin\verify.dll
0x6d3e0000 - 0x6d3ff000 	C:\Program Files (x86)\SmartFoxServerPRO_1.6.6\jre\bin\java.dll
0x6d8a0000 - 0x6d8af000 	C:\Program Files (x86)\SmartFoxServerPRO_1.6.6\jre\bin\zip.dll
0x69d10000 - 0x69d45000 	C:\WINDOWS\system32\rsaenh.dll
0x76920000 - 0x769e2000 	C:\WINDOWS\system32\USERENV.dll
0x6d6c0000 - 0x6d6d3000 	C:\Program Files (x86)\SmartFoxServerPRO_1.6.6\jre\bin\net.dll
0x71c00000 - 0x71c17000 	C:\WINDOWS\system32\WS2_32.dll
0x71bf0000 - 0x71bf8000 	C:\WINDOWS\system32\WS2HELP.dll
0x7db30000 - 0x7dbb0000 	C:\WINDOWS\System32\mswsock.dll
0x77670000 - 0x777a9000 	C:\WINDOWS\syswow64\ole32.dll
0x76ed0000 - 0x76efa000 	C:\WINDOWS\system32\DNSAPI.dll
0x76f70000 - 0x76f77000 	C:\WINDOWS\System32\winrnr.dll
0x76f10000 - 0x76f3e000 	C:\WINDOWS\syswow64\WLDAP32.dll
0x76f80000 - 0x76f85000 	C:\WINDOWS\system32\rasadhlp.dll
0x6d6e0000 - 0x6d6e9000 	C:\Program Files (x86)\SmartFoxServerPRO_1.6.6\jre\bin\nio.dll
0x6a230000 - 0x6a28a000 	C:\WINDOWS\system32\hnetcfg.dll
0x71ae0000 - 0x71ae8000 	C:\WINDOWS\System32\wshtcpip.dll
0x76cf0000 - 0x76d0a000 	C:\WINDOWS\system32\iphlpapi.dll
0x76cd0000 - 0x76ce9000 	C:\WINDOWS\system32\MPRAPI.dll
0x76df0000 - 0x76e24000 	C:\WINDOWS\system32\ACTIVEDS.dll
0x76dc0000 - 0x76de8000 	C:\WINDOWS\system32\adsldpc.dll
0x76b80000 - 0x76bae000 	C:\WINDOWS\system32\credui.dll
0x7c8d0000 - 0x7d0cf000 	C:\WINDOWS\syswow64\SHELL32.dll
0x6a6c0000 - 0x6a712000 	C:\WINDOWS\syswow64\SHLWAPI.dll
0x76a80000 - 0x76a92000 	C:\WINDOWS\system32\ATL.DLL
0x6a720000 - 0x6a7ab000 	C:\WINDOWS\syswow64\OLEAUT32.dll
0x76e30000 - 0x76e3c000 	C:\WINDOWS\system32\rtutils.dll
0x7e020000 - 0x7e02f000 	C:\WINDOWS\system32\SAMLIB.dll
0x770e0000 - 0x771e8000 	C:\WINDOWS\system32\SETUPAPI.dll
0x7dbd0000 - 0x7dcd3000 	C:\WINDOWS\WinSxS\WOW64_Microsoft.Windows.Common-Controls_6595b64144ccf1df_6.0.3790.3959_x-ww_5FA17F4E\comctl32.dll

VM Arguments:
jvm_args: -Xms64M -Xmx1500M -Dfile.encoding=UTF-8 -Djava.util.logging.config.file=logging.properties 
java_command: it.gotoandplay.smartfoxserver.SmartFoxServer
Launcher Type: SUN_STANDARD

Environment Variables:
PATH=C:\Program Files (x86)\PHP\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files\TortoiseSVN\bin
USERNAME=XXXXXXXX
OS=Windows_NT
PROCESSOR_IDENTIFIER=EM64T Family 6 Model 23 Stepping 7, GenuineIntel



---------------  S Y S T E M  ---------------

OS: Windows Server 2003 family Build 3790 Service Pack 2

CPU:total 4 (4 cores per cpu, 1 threads per core) family 6 model 7 stepping 7, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3

Memory: 4k page, physical 4192284k(2346116k free), swap 4194303k(4194303k free)

vm_info: Java HotSpot(TM) Server VM (11.3-b02) for windows-x86 JRE (1.6.0_13-b03), built on Mar  9 2009 01:10:11 by "java_re" with MS VC++ 7.1

time: Thu Jun 17 16:29:26 2010
elapsed time: 83014 seconds

We are running smartfox 1.6.8.

If you could shed any light on our problem that would be great!

Thank you.

Post by **Lapo** » 21 Jun 2010, 06:32

Hello, pay attention to your server side code, evidently you are constantly leaking data to the point where no more memory can be allocated.

My first guess would be to double check the usage of Threads/setIntervals especially when used in Room Level extensions. Always make sure to get rid of those in the destroy() method of your server code.

Also please monitor the Server via the Admin Tool to see the thead count and general statuts of the Server queues.
If this is not sufficient you should use VisualVM in the SFS JVM to see the runtime status and allocation data of the memory.

VisualVM is extremely simple to use and very helpful in cases like these.
Download here: https://visualvm.dev.java.net/

Mather · Post by **Mather** » 21 Jun 2010, 13:25

Thanks I will download that VisualVM tool and see what i can find!

Mather · Post by **Mather** » 15 Jul 2010, 09:01

We have been using teh VisualVM tool to monitor the status of the server when the problems ahve been occuring but still can't find the problem. I've attached two graphs of the state of the VM when the crashes occur. The odd thing is that they seem to crash with very little memory being used, less than 100mb in fact even tho the max value is over 1gb. As for the other parts to look out for our game is running 5 Zones each with 1 zone level extension, no room level ones and the crashes have been occuring while only 1 zone is being occupied. Also the code doesnt use any setInterval codes and just uses one scheduler thread.

Any help you can provide to our situation would be greatly appreciated thankyou!

Crash 1: http://94.198.81.36/stuff/crash1.png

Crash 2: http://94.198.81.36/stuff/crash2.png

Edit: I forgot to add we got a slightly different message for the last crash but not sure if it means anything different:

Code: Select all

Exception in thread "selector" java.lang.OutOfMemoryError
	at sun.misc.Unsafe.allocateMemory(Native Method)
	at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:99)
	at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
	at it.gotoandplay.smartfoxserver.util.Attachment.<init>(Attachment.java:33)
	at it.gotoandplay.smartfoxserver.SmartFoxServer.checkNewConnections(SmartFoxServer.java:1311)
	at it.gotoandplay.smartfoxserver.EventReader.run(EventReader.java:31)
	at java.lang.Thread.run(Thread.java:619)

Post by **Lapo** » 15 Jul 2010, 16:38

Threads are under control, which is good.
The heap also doesn't look very busy as it barely reaches the ceiling of 100Mb and the usage pattern is not suggestive of a memory leak.

The first crashed you posted was very low level, probably the JVM wanted to allocate more memory but the OS didn't seem to like it.

The second error is related with the Direct Buffers which are allocated in a completely different memory area.

There's a JVM Setting:

Code: Select all

-XX:MaxDirectMemorySize=<size>

which allows to add more memory for Direct Buffer. By default this should be 64M and you could try to grow it to 128M or 256M

Mather · Post by **Mather** » 16 Jul 2010, 08:21

Brilliant, thankyou we will try this.

Mather · Post by **Mather** » 21 Jul 2010, 10:21

5 days and no crashes

.

Thanks for your help that seems to have solved our problems!