Well I actually figured out what it was.
My client sends health reports every 2 minutes or so. The MaxIdleTime is set to 5 minutes so I knew it wasn't that.
However, I read on the forums that ELB cuts the tcp connection after 1 minute of inactivity, so I had to switch my health reports to every ...