Technical Inquiry: Deployment Strategy (Dockerization & CI/CD) for SmartFoxServer

[hieuitvnsoft]

Dear SmartFox Support Team,

We are reaching out to seek technical guidance regarding the optimal deployment and continuous integration/continuous deployment (CI/CD) strategy for our SmartFoxServer (SFS) environment.

Our goal is to significantly accelerate our product server deployment process and enhance our disaster recovery/backup plan. We have two main areas of inquiry:

1. Dockerization of SmartFoxServer

Is it technically feasible and officially supported to build and deploy SmartFoxServer, along with its custom Extension files, within a Docker container?

We believe containerization would greatly help us streamline our deployment pipeline and simplify the overall management and backup of the server product stack.

2. License Management and CI/CD Solutions

Based on your answer to the first question, we require solutions for integrating crucial configurations:

A. If Dockerization is feasible:
What are the recommended best practices for securely integrating the SmartFox license file and the core configuration files (e.g., config.xml) into the Docker image or container environment? Specifically, should we be using Docker Volumes, environment variables, or other secret management solutions for these critical assets?

B. If Dockerization is NOT recommended/supported:
Could you advise on a secure and robust CI/CD workflow strictly for deploying our compiled Extension files (e.g., .jar, .zone.xml) onto an existing, running SmartFoxServer instance? We are looking for a method that ensures security and minimizes downtime during updates.

Thank you very much for your time and comprehensive assistance with these questions. Your recommendations will be crucial in shaping our infrastructure roadmap.

Best regards,

[Lapo]

Hi,

Is it technically feasible and officially supported to build and deploy SmartFoxServer, along with its custom Extension files, within a Docker container?

Yes. SmartFoxServer is a Java application and as such can run virtualized as any other similar apps.
However I'd like to better understand the use case for using Docker and multiple SFS instances. In most scenarios running SmartFoxServer on a single machine (or a small fixed number of them) is more than enough, and much easier to manage.

SmartFoxServer has great vertical scalability and can handle tens of thousands of CCU on a single machine (even a 8-core machine with a 4-8GB RAM can be enough), depending on the use case. Even if you need more than one machine for your deployment it may be that just 2-3 instances is all you need. (For more info see here: https://www.smartfoxserver.com/download ... X-100K.pdf)

I mention this because it's a simpler architecture to build than going full-on with virtualization or micro-service style, where you need to deal with all the extra complexity of dynamic deployments, load balacing etc.

Is this something you have considered?
Thanks

[hieuitvnsoft]

Hi,

Is it technically feasible and officially supported to build and deploy SmartFoxServer, along with its custom Extension files, within a Docker container?

Yes. SmartFoxServer is a Java application and as such can run virtualized as any other similar apps.
However I'd like to better understand the use case for using Docker and multiple SFS instances. In most scenarios running SmartFoxServer on a single machine (or a small fixed number of them) is more than enough, and much easier to manage.

SmartFoxServer has great vertical scalability and can handle tens of thousands of CCU on a single machine (even a 8-core machine with a 4-8GB RAM can be enough), depending on the use case. Even if you need more than one machine for your deployment it may be that just 2-3 instances is all you need. (For more info see here: https://www.smartfoxserver.com/download ... X-100K.pdf)

I mention this because it's a simpler architecture to build than going full-on with virtualization or micro-service style, where you need to deal with all the extra complexity of dynamic deployments, load balacing etc.

Is this something you have considered?
Thanks

Thank you very much for your detailed and helpful response regarding the significant vertical scalability of SmartFoxServer. We appreciate the insight that running a few powerful instances is often simpler and sufficient for high CCU numbers.

We would like to clarify our core motivation for exploring Dockerization:

Our primary concern is CI/CD automation and safe deployment of our custom SFS Extensions to our product server environment.

The Need for Automated Rollback

While vertical scaling is impressive, our main challenge lies in eliminating manual deployment steps and ensuring robust failure handling during Extension updates.

We are investigating Docker because it provides immutable infrastructure. By packaging the SmartFox Server base and a specific, tested Extension version into a single container image, we gain the ability to:

Automate Deployment: Easily and consistently deploy a new version to one or many VMs simultaneously.

Instant Rollback: If a new Extension version introduces a critical bug, we can immediately and safely revert to the previous stable version simply by redeploying the container image with the older tag. This rollback mechanism is far quicker and more reliable than manually swapping out Extension files on a live server.

Simplified Horizontal Scaling: Should we need to scale out to multiple VMs in the future, a pre-built Docker image simplifies that horizontal deployment considerably, as every new instance is guaranteed to be identical.

Given this context—that our goal is primarily focused on achieving safe, automated, and repeatable CI/CD deployment with guaranteed rollback capability—we would greatly appreciate your best practices advice:

Could you please offer your recommendations on the most secure and robust way to implement this automated deployment strategy, regardless of whether it uses Docker or an alternative recommended by SmartFox that fulfills the same requirements for automation and safe rollback?

We are particularly interested in the secure handling of the license and configuration files within this automated pipeline.

Thank you again for your continued support.

Best regards,

[Lapo]

I have a few further questions about some of the points you have highlighted:

Instant Rollback: If a new Extension version introduces a critical bug, we can immediately and safely revert to the previous stable version simply by redeploying the container image with the older tag. This rollback mechanism is far quicker and more reliable than manually swapping out Extension files on a live server.

Redeploying a number of instances for a rollback means killing all the active connections from all those servers, which is quite impactful. A less disruptive operation would be to only re-deploy the Extension jar files in each server. This would trigger a hot-reload of the code without affecting the current players connections.

Granted, this may not always be possible depending on how the application/game works. Example: if the game state data format is affected by the re-deploy it may be necessary to stop all current games before applying the changes. This mainly depends on the app design and its features. Keeping the re-deploy aspect in mind during development can be useful to make the code more decoupled and less prone to issues when reloaded at runtime.

Simplified Horizontal Scaling: Should we need to scale out to multiple VMs in the future, a pre-built Docker image simplifies that horizontal deployment considerably, as every new instance is guaranteed to be identical.

Horizontal scaling is never simple because of load balacing and "split-brain" syndrome.

1) Load Balancing
SmartFoxServer works with persistent connections (TCP or websocket) and as such is not as easily load balanced as a web application (this is true for any game server, btw). You cannot just stick a layer 7 load balancer in front of a bunch of SmartFox instances and call it a day.
See here for more details:
https://www.smartfoxserver.com/download ... ecture.pdf
https://smartfoxserver.com/blog/load-ba ... server-2x/

2) Split-brain syndrome
When running a distributed system you'll likely need some kind of "shared state" across all servers to be able to coordinate them.
Example: a main Lobby server will likely need to know about all the game servers available to route players to the right instance when they want to play.
This could be handled by a central database of some kind (Redis, Hazelcast or similar) but you also have to factor in all the Docker VMs coming and going at runtime and how to co-ordinate them with the load balancing and overall state of the distributed application.
We have built several large clustered system like these and we usually recommend this approach only to large companies that have the budget and expertise to handle it.

Just to be clear I am not trying to dissuade you from pursuing the kind of architecture you have described, especially if you have the expertise to implement it.
However if you're starting out on a new project and you're kind of new to distributed multiplayer in general, there are much more approachable solutions that can get you quite far and save you a lot of headaches and money.

For instance, you could still implement a CI/CD workflow but using a simpler production environment that works with a fixed number of servers and makes the whole management easier. If you want resilience you can keep an active backup copy of the production servers that could take over in case of a major server failure.

If you're worried about sudden traffic spikes from the game becoming super popular: keep in mind that you can easily achieve a capacity of 50K-100K CCU with a simpler system like the one I have described. This means millions of active users a day (DAU), these are not small numbers. And in case you'll find yourselves with a constant player base growth you will have enough time to think of the next version of your application running in a fully containerized and automated environment that could scale even higher.

Hope everything was clear. If not let us know.
Cheers

[hieuitvnsoft]

I have a few further questions about some of the points you have highlighted:

Instant Rollback: If a new Extension version introduces a critical bug, we can immediately and safely revert to the previous stable version simply by redeploying the container image with the older tag. This rollback mechanism is far quicker and more reliable than manually swapping out Extension files on a live server.

Redeploying a number of instances for a rollback means killing all the active connections from all those servers, which is quite impactful. A less disruptive operation would be to only re-deploy the Extension jar files in each server. This would trigger a hot-reload of the code without affecting the current players connections.

Granted, this may not always be possible depending on how the application/game works. Example: if the game state data format is affected by the re-deploy it may be necessary to stop all current games before applying the changes. This mainly depends on the app design and its features. Keeping the re-deploy aspect in mind during development can be useful to make the code more decoupled and less prone to issues when reloaded at runtime.

Simplified Horizontal Scaling: Should we need to scale out to multiple VMs in the future, a pre-built Docker image simplifies that horizontal deployment considerably, as every new instance is guaranteed to be identical.

Horizontal scaling is never simple because of load balacing and "split-brain" syndrome.

1) Load Balancing
SmartFoxServer works with persistent connections (TCP or websocket) and as such is not as easily load balanced as a web application (this is true for any game server, btw). You cannot just stick a layer 7 load balancer in front of a bunch of SmartFox instances and call it a day.
See here for more details:
https://www.smartfoxserver.com/download ... ecture.pdf
https://smartfoxserver.com/blog/load-ba ... server-2x/

2) Split-brain syndrome
When running a distributed system you'll likely need some kind of "shared state" across all servers to be able to coordinate them.
Example: a main Lobby server will likely need to know about all the game servers available to route players to the right instance when they want to play.
This could be handled by a central database of some kind (Redis, Hazelcast or similar) but you also have to factor in all the Docker VMs coming and going at runtime and how to co-ordinate them with the load balancing and overall state of the distributed application.
We have built several large clustered system like these and we usually recommend this approach only to large companies that have the budget and expertise to handle it.

Just to be clear I am not trying to dissuade you from pursuing the kind of architecture you have described, especially if you have the expertise to implement it.
However if you're starting out on a new project and you're kind of new to distributed multiplayer in general, there are much more approachable solutions that can get you quite far and save you a lot of headaches and money.

For instance, you could still implement a CI/CD workflow but using a simpler production environment that works with a fixed number of servers and makes the whole management easier. If you want resilience you can keep an active backup copy of the production servers that could take over in case of a major server failure.

If you're worried about sudden traffic spikes from the game becoming super popular: keep in mind that you can easily achieve a capacity of 50K-100K CCU with a simpler system like the one I have described. This means millions of active users a day (DAU), these are not small numbers. And in case you'll find yourselves with a constant player base growth you will have enough time to think of the next version of your application running in a fully containerized and automated environment that could scale even higher.

Hope everything was clear. If not let us know.
Cheers

Thank you for the detailed explanation and the important warnings regarding the complexities of horizontal scaling, load balancing, and the "split-brain" syndrome. We fully understand that distributed multiplayer architecture is non-trivial and often best reserved for very large deployments.

We would like to confirm that horizontal scaling is currently not our primary objective or requirement. We intend to stick to a fixed, small number of vertically scaled, robust VMs, as you recommended, which is more than sufficient for our current CCU targets (50K-100K+).

Clarifying the Core Goal: Automation, Immutability, and Disaster Recovery

Our decision to explore Docker is solely based on improving our DevOps efficiency and disaster recovery capabilities. We are looking for a process that provides:

Immutability: Each deployable unit (the Docker image) contains a known, tested version of the SmartFox base and the Extension code.

Rapid Deployment/Rollback: The ability to swap out a faulty version for a stable one, or deploy a new production instance, by simply running a Docker command (or deployment tool) in minutes, dramatically reducing potential downtime from hours to minutes.

Product Isolation: Given we manage multiple products, using dedicated Docker images for each product allows us to easily isolate environments and deploy a specific server stack to a new VM instantly when needed.

Since you confirmed that SmartFoxServer can run virtualized as a Java application, we would like to proceed with testing this approach for our CI/CD pipeline.

To help us begin prototyping the deployment process, could you please provide a few best practice configurations or a small demo of a Dockerfile setup, specifically addressing the secure integration and configuration of the following critical elements:

Network Ports: Recommended settings for mapping and exposing standard SmartFox ports (Socket, WebSocket, and potentially SSL).

SSL/TLS: The best way to inject and configure the necessary key files for SSL/TLS within the container environment.

License Management: The secure method for inserting the SmartFox license file into the container, perhaps using a bind mount (Docker Volume) or secrets management, as these files cannot be baked directly into the image.

Any guidance or sample configuration for these initial steps would be extremely valuable to us.

Thank you once again for focusing on our specific CI/CD needs.

Best regards,

[Lapo]

To help us begin prototyping the deployment process, could you please provide a few best practice configurations or a small demo of a Dockerfile setup, specifically addressing the secure integration and configuration of the following critical elements:

I am not a Docker expert, so I cannot guide you with a specific Docker configuration. I recommend you refer to their official documentation for all the details.

Network Ports: Recommended settings for mapping and exposing standard SmartFox ports (Socket, WebSocket, and potentially SSL).

At the bottom of this document you can find the list of ports used by SmartFoxServer.
https://docs2x.smartfoxserver.com/Getti ... stallation
These values are not mandatory and can be changed, if required.

SSL/TLS: The best way to inject and configure the necessary key files for SSL/TLS within the container environment.

You can check our documentation here for a step by step guide on how to build a keystore containing your certificate.
https://docs2x.smartfoxserver.com/Getti ... yptography

License Management: The secure method for inserting the SmartFox license file into the container, perhaps using a bind mount (Docker Volume) or secrets management, as these files cannot be baked directly into the image.

The license consists of a code that is plugged in the server via AdminTool. For an automated system you will have to put the license code under the SFS2X/config/server.xml file
By default there's an empty <licenseCode/> tag that you can populate with the actual code. Example:

Code: Select all

<licenseCode>ABC123abc</licenseCode>

A Docker script can be used to automate the insertion of the code at the right spot in the file. Or you can pre-populate the license code for each Docker image, if it is convenient. Keep in mind: every instance will run with a different license code.

Hope it helps

[hieuitvnsoft]

To help us begin prototyping the deployment process, could you please provide a few best practice configurations or a small demo of a Dockerfile setup, specifically addressing the secure integration and configuration of the following critical elements:

I am not a Docker expert, so I cannot guide you with a specific Docker configuration. I recommend you refer to their official documentation for all the details.

This detailed information is exactly what we needed to begin prototyping our Docker-based CI/CD workflow for automated deployment and rapid rollback. We will now proceed with referring to the official documentation and start our implementation trials.

We appreciate the time you took to address our specific needs. Thanks