Hundreds of MalformedInputException warnings per hour

torncanvas · Post by **torncanvas** » 18 Dec 2008, 20:29

I've been looking on the forums to see what this error is:

java.nio.charset.MalformedInputException: Input length = 1
        at java.nio.charset.CoderResult.throwException(CoderResult.java:260)
        at java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:781)
        at it.gotoandplay.smartfoxserver.SmartFoxServer.readIncomingMessages(SmartFoxServer.java:1024)
        at it.gotoandplay.smartfoxserver.EventReader.run(EventReader.java:32)
        at java.lang.Thread.run(Thread.java:619)

We released open beta yesterday and have between 150 and 350 concurrent users actively playing the game at any given time. On average, we see this error every few seconds.

You've warned in other threads that there could be a serious issue if this is happening hundreds of times per hour, and it is for us. Just so you know, our server extension is in Java, and I don't think there are a lot of Linux clients connecting.

Any suggestions for helping to find out more about this error?

We've had several other problems related to rooms hanging, connecting issues, logging in, not loading SWFs properly, etc. On a few occasions, we've ended up having to write our own code to replace some Smartfox functionality that didn't seem to be working properly. Based on this experience, we don't think this error is related to hacking or Linux clients, but more likely some sort of communication problem with either our code or Smartfox's code.

Post by **Lapo** » 19 Dec 2008, 09:29

You've warned in other threads that there could be a serious issue if this is happening hundreds of times per hour, and it is for us. Just so you know, our server extension is in Java, and I don't think there are a lot of Linux clients connecting.

The exception is fired everytime a client is sending a request that the server cannot understand. If this happens it means that the client talking to SFS is not using the SFS API.

If for example your SFS instance runs on port 80 many web spiders might assume that a web server is listening, they send an HTTP request and they don't get a response back, plus SFS complains because it doesn't recognize the request. Makes sense?

Another option is that your machine is under a Denail Of Service (DOS) attack. You should monitor your connections by using netstat or similar tools. Also the Admin Tool will help by telling you how many socket connections vs logged users you have.
You shouldn't notice a huge discrepancy... you would normally expect to have 10-20% more connections than logged users.

We've had several other problems related to rooms hanging, connecting issues, logging in, not loading SWFs properly, etc.

Each of these problems should be explained in detail as it can depend on many factors which range from the server config, to your hardware specs, available bandwidth and whatnot.

In our documentation we provide many guidelines for proper server configuration, advanced settings, jvm tuning and general troubleshooting.
As regards troubleshooting we have a document that will help you here -> http://www.smartfoxserver.com/docs/docP ... ooting.htm

And btw... these problems are normally showing up in betas

That's where you finally see the application running under real user traffic, including user's misbehavior, slow connections, network lag etc...
With a little patience you will be able to fix your code and fine tune the server

On a few occasions, we've ended up having to write our own code to replace some Smartfox functionality that didn't seem to be working properly.

Can you give us all the details?
No software in the world is bugfree, and so aren't our API however they have been around for quite a long time and improved on a regular basis and we're not currently aware of particular problems such as "no working properly"

torncanvas · Post by **torncanvas** » 19 Dec 2008, 17:59

Thanks for offering to help with the other stuff. We should have a phone call at some point to go over it, as it would be time-consuming to have a text-based conversation about it.

My point was just that we have a good amount of our own code going on, which would suggest that there's more chance that this error could occur than in a game that uses a more templated integration of SFS.

Currently we have 177 users and 179 socket connections, and our server is running on port 9339.

Regarding the MalformedInputException, I checked the logs of our dev server. That server is only accessible from an authenticated website for development purposes, which people from our company and Kongregate have access to. I have found this error in some previous logs when we were testing more heavily, and it was not showing up in the more recent logs when we haven't been testing the game much. This suggests that it is not a DoS issue, due to the security (Kong is good about that) and limited access to the dev site, and that it occurs more frequently when we're using the game more often.

Is it possible that some of our messages are getting cut off due to imperfect network connections? Or maybe we're inadvertently sending messages to the server somehow through a means other than the API? I'll keep looking into this issue, and some Kong staff can watch network traffic, but so far all signs point to this being related to how we're using SFS.

Post by **Lapo** » 19 Dec 2008, 22:08

Currently we have 177 users and 179 socket connections, and our server is running on port 9339.

This looks very good.

Is it possible that some of our messages are getting cut off due to imperfect network connections?

Nope

Or maybe we're inadvertently sending messages to the server somehow through a means other than the API?

Sounds strange, but if you do you should know it? Right?

Are those messages all the same? (Input length = 1)
In that case it would seem that "something" is connecting and sending a control character? Just 1 byte according to the error report

Other causes might be a firewall in between shutting down messages prematurely... but it's just an idea.

Have you changed anything recently?

torncanvas · Post by **torncanvas** » 19 Dec 2008, 22:43

A large majority of the messages are of length 1, yes. Every now and then, we'll get a length of 2, 3, or 4.

The firewall issue is an interesting point.

Also, we've checked on another Kong premium game, and they seem to get the errors, too. I'm not sure what that means, maybe that's evidence that we're not sending messages using something other than the API.

By the way, thanks for helping out with this so far. We appreciate it, and I hope we can get it resolved or at least understood sometime soon. Merry Christmas.

Post by **Lapo** » 20 Dec 2008, 18:40

You too

Boing · Post by **Boing** » 15 Jan 2010, 10:08

Any insights this?
We also see lots of "Unexpected / Malformed input: java.nio.charset.MalformedInputException: Input length = 1":

INFO | jvm 1 | 2010/01/15 10:48:19 | 581 WARN [selector] SFSLog4JLogger.run (32): SFS: Unexpected / Malformed input: java.nio.charset.MalformedInputException: Input length = 1
INFO | jvm 1 | 2010/01/15 10:48:19 | java.nio.charset.MalformedInputException: Input length = 1
INFO | jvm 1 | 2010/01/15 10:48:19 | at java.nio.charset.CoderResult.throwException(CoderResult.java:260)
INFO | jvm 1 | 2010/01/15 10:48:19 | at java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:781)
INFO | jvm 1 | 2010/01/15 10:48:19 | at it.gotoandplay.smartfoxserver.SmartFoxServer.readIncomingMessages(SmartFoxServer.java:1017)
INFO | jvm 1 | 2010/01/15 10:48:19 | at it.gotoandplay.smartfoxserver.EventReader.run(EventReader.java:32)
INFO | jvm 1 | 2010/01/15 10:48:19 | at java.lang.Thread.run(Thread.java:619)
INFO | jvm 1 | 2010/01/15 10:48:19 | 589 WARN [selector] SFSLog4JLogger.run (32): SFS: Unexpected / Malformed input: java.nio.charset.MalformedInputException: Input length = 1
INFO | jvm 1 | 2010/01/15 10:48:19 | java.nio.charset.MalformedInputException: Input length = 1
INFO | jvm 1 | 2010/01/15 10:48:19 | at java.nio.charset.CoderResult.throwException(CoderResult.java:260)
INFO | jvm 1 | 2010/01/15 10:48:19 | at java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:781)
INFO | jvm 1 | 2010/01/15 10:48:19 | at it.gotoandplay.smartfoxserver.SmartFoxServer.readIncomingMessages(SmartFoxServer.java:1017)
INFO | jvm 1 | 2010/01/15 10:48:19 | at it.gotoandplay.smartfoxserver.EventReader.run(EventReader.java:32)
INFO | jvm 1 | 2010/01/15 10:48:19 | at java.lang.Thread.run(Thread.java:619)

Since we have a realtime action game we send a lot of small packets using the raw protocol.
The server is running fine, the exceptions don't seem to cause any problems, but they are nasty because our log files are filled with non-critical exceptions and stack traces which makes it difficult to catch real problems quickly.
In this sense it would already help us if SFS would not log like this to standard out...

Post by **Lapo** » 15 Jan 2010, 11:50

In addition to the explanation provided here there is also another one that should be known.
The Flash Player for Linux has a bug at the socket level which could generate this issue. Adobe knows this but is taking forever to fix it. If you have players under Linux you might see those log multiply.
Good news is that we have worked around the issue in the latest releases. See the latest download -> SFS PRO patch 1.6.8

Detailed discussion of the problem are found here:
http://forums.smartfoxserver.com/viewtopic.php?t=6358
http://forums.smartfoxserver.com/viewtopic.php?t=6200

Evgeny · Post by **Evgeny** » 11 May 2010, 18:22

I'm also getting huge number of the same warnings, input length is always 1. Server is SFS PRO 1.6.8, it is running on port 9339 and usually there are more than 500 users online. According to Admin tool, number of socket connections is equal to the number of online users. Only 0.04% of users are using Linux, so I don't think it is related to the bug in Linux Flash player. Also client is compiled using SFS API 1.6.1 and it doesn't use raw protocol, only JSON.
Is it possible to get more information about this malformed input?
For example, trace it. I know that one symbol probably won't give too much hints, but nevertheless. Also it would be good to show ip address of sender.

Code: Select all

INFO   | jvm 1    | 2010/05/11 20:27:08 | 20:27:08.837 - [ WARNING ] > Unexpected / Malformed input: java.nio.charset.MalformedInputException: Input length = 1
INFO   | jvm 1    | 2010/05/11 20:27:08 | java.nio.charset.MalformedInputException: Input length = 1
INFO   | jvm 1    | 2010/05/11 20:27:08 |       at java.nio.charset.CoderResult.throwException(CoderResult.java:260)
INFO   | jvm 1    | 2010/05/11 20:27:08 |       at java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:781)
INFO   | jvm 1    | 2010/05/11 20:27:08 |       at it.gotoandplay.smartfoxserver.SmartFoxServer.readIncomingMessages(SmartFoxServer.java:1017)
INFO   | jvm 1    | 2010/05/11 20:27:08 |       at it.gotoandplay.smartfoxserver.EventReader.run(EventReader.java:32)
INFO   | jvm 1    | 2010/05/11 20:27:08 |       at java.lang.Thread.run(Thread.java:619)

Post by **Lapo** » 12 May 2010, 08:19

It could be a hacker attack but if you say that sockets and users are almost the same number it's probably not.
Linux users is the 2nd logical explanation, even just 10-20 of them could make the server fire many of those warnings (which btw are harmless besides bloating the logs a bit)

Is it possible to get more information about this malformed input?

Please read the links posted in my previous post

Evgeny · Post by **Evgeny** » 12 May 2010, 09:55

Lapo wrote:Please read the links posted in my previous post

I did, but I really don't think it is related to Linux users. Our client is complied using latest SFS API. And there's a check to make sure that user is using latest version of the game client, so it cannot be that these Linux users are using old game client due to browser caching.
Is it possible to trace IP address of the sender when malformed exception happens?

Post by **Lapo** » 12 May 2010, 15:03

At the moment no

We'll add it in the next update.

Evgeny · Post by **Evgeny** » 12 May 2010, 15:14

Lapo wrote:At the moment no We'll add it in the next update.

Ok, thank you.