forums.smartfoxserver.com

Posted: **18 Jan 2025, 14:02**

I have some pregenerated random data like noisemaps or lookup tables.
One lookup table is 4096 * 4096 short elements so thats abt 65.5million shorts which comes down to 32MB of data that I read from a binary file into a 2D array short[4096][4096].
Loading this lookup table into smartfox memory takes about 80-90 seconds, which is very long to read some binary...

I have another noisemap which is actualy 8192*8192 short elements, so thats abt 260m short elements which is about 128MB of data again in a binary file, taht I also read into a 2D array short[][]. This step takes abt 4x as long(obvious why) so abt 5 minutes.

that means startup of the server can take 6 minutes just for loading some pregenerated memory from the hard disk or SSD into app memory...

Is this normal?
I see SFS is using the max number of threads (59) and RAM is maxed out at 4GB(not a lot but should be more than enough to load 150MB of data?)
also CPU is maxxed out aswell.

What seetings can i potentially change to make this faster, OR is there a better way to load data onto the server at startup?

Thanks.

Attached: screenshot of terminal and admin panel.

LoadingNoisemapServer.png: (146.64 KiB) Not downloaded yet

Posted: **18 Jan 2025, 14:06**

Code: Select all

package manager;

import java.io.DataInputStream;
import java.io.FileInputStream;
import java.io.IOException;

import data.Fixed;
import lib.LibExtension;

public class PersistentDataManager
{
    static LibExtension _ext;

    public static void Init(LibExtension ext)
    {
        _ext = ext;
    }

    public static Fixed[][] LoadNoiseMapFromBinary(String filePath, int size)
    {
        Fixed[][] noiseMap = new Fixed[size][];
        try (FileInputStream fileInputStream = new FileInputStream(filePath);
                DataInputStream dataInputStream = new DataInputStream(fileInputStream))
        {

            for (int y = 0; y < size; y++)
            {
                noiseMap[y] = new Fixed[size];
                for (int x = 0; x < size; x++)
                {
                    // Read two bytes as little-endian and construct the short value
                    int byte1 = dataInputStream.readUnsignedByte(); // Least significant byte
                    int byte2 = dataInputStream.readUnsignedByte(); // Most significant byte
                    short value = (short) ((byte2 << 8) | byte1); // Combine bytes into short

                    // Construct the Fixed object using the short value
                    noiseMap[y][x] = new Fixed(value, true);
                }
            }
        }

        catch (IOException e)
        {
            _ext.trace("Error loading noise maps: " + e.getMessage());
            e.printStackTrace();
        }

        _ext.trace("Loaded noisemap from: " + filePath);
        return noiseMap;
    }

}

Notes:
1.Fixed is a fixed point number that uses a short.
2.Csharp writes binary in little endian, so reading a short in java requires us to switch the bytes around since java only reads in big-endian.

Posted: **18 Jan 2025, 14:37**

Im guessing its also because of my Fixed class getting instantiated a few hundred million times, thats a lot of memory overhead due to padding.
I might switch it to an actual short[][] instead of Fixed[][] to see if that helps, but it will make the whole fixed number system more complex and error prone as there is no more Fixed wrapper class around the short values...

Posted: **18 Jan 2025, 16:29**

So I switched it from the class Fixed to suing a pure raw short[][] array, that solved the memory issue, but it looks like the sfs process is capped at 15-20% of CPU, or maybe EXTWorker2 is capped to 20% of CPU. So I guess I have to make this binary reading system concurrent so that it uses more threads?

EDIT: Since EXTWorker2 is using around 16.67% of the total CPU im gonna say its capped to that value because theres 6 main threads that are capped at that value which makes 100%. :wink:

Attached:
1. Terminal and Admin panel during reading binary.
2. Terminal and Admin panel after reading to calculate total time it took (about 7-8 minutes to load 2 binary files
- abt 150mb of data into short[][] arrays.

usingShorts1.png: (143.62 KiB) Not downloaded yet

using shorts2.png: (152.22 KiB) Not downloaded yet

Is it possible to make the following run concurrently using multiple EXTWorker threads somehow?

Code: Select all

package manager;

import java.io.DataInputStream;
import java.io.FileInputStream;
import java.io.IOException;

import lib.LibExtension;

public class PersistentDataManager
{
    static LibExtension _ext;

    public static void Init(LibExtension ext)
    {
        _ext = ext;
    }

    public static short[][] LoadNoiseMapFromBinary(String filePath, int size)
    {
        short[][] noiseMap = new short[size][];
        try (FileInputStream fileInputStream = new FileInputStream(filePath);
                DataInputStream dataInputStream = new DataInputStream(fileInputStream))
        {

            for (int y = 0; y < size; y++)
            {
                noiseMap[y] = new short[size];
                for (int x = 0; x < size; x++)
                {
                    // Read two bytes as little-endian and construct the short value
                    int byte1 = dataInputStream.readUnsignedByte(); // Least significant byte
                    int byte2 = dataInputStream.readUnsignedByte(); // Most significant byte
                    short value = (short) ((byte2 << 8) | byte1); // Combine bytes into short

                    noiseMap[y][x] = value;
                    // Construct the Fixed object using the short value
                    // noiseMap[y][x] = new Fixed(value, true);
                }
            }
        }

        catch (IOException e)
        {
            _ext.trace("Error loading noise maps: " + e.getMessage());
            e.printStackTrace();
        }

        _ext.trace("Loaded noisemap from: " + filePath);
        return noiseMap;
    }

}

Posted: **18 Jan 2025, 17:03**

Hi,
there is no "capping" of threads anywhere. It must be the I/O operation that doesn't fully use a single thread.
Also SFS has little to no influence on this task since you're just using Java's own API to perform the loading.

Anyways check this article, I think it will help:
https://lemire.me/blog/2019/03/18/dont- ... m-a-straw/

Cheers

Posted: **18 Jan 2025, 19:48**

Lapo wrote:Hi,
there is no "capping" of threads anywhere. It must be the I/O operation that doesn't fully use a single thread.
Also SFS has little to no influence on this task since you're just using Java's own API to perform the loading.

Anyways check this article, I think it will help:
https://lemire.me/blog/2019/03/18/dont- ... m-a-straw/

Cheers

Hi,

You are a boss Lapo. It now takes less than 1 second to load both noisemaps! WOW!
Using the ByteBuffer as you suggested.

Cheers!

Posted: **18 Jan 2025, 20:12**

Using the MappedByteBuffer does seem to introduce a big allocation in memory which doesnt get cleaned up by the GC, I could use reflection to call the cleaner/clean object directly but depending on the JVM reflective access is prohibited so this probably wont be a good idea on a production server...
I read that it gets stored in direct memory aswell not on the heap, so it will only be collected by the GC on a full pass, which I can call but even then its not guaranteed to happen immediately by using

Code: Select all

System.gc();

.

If I could force the gc to do a complete pass that would be a good solution maybe, since loading this data is only done once at startup and with 0 users connected, so doing a full gc then before letting any users connect would actually be fine.

Any other options I can use, besides just waiting for the GC to hopefully sometime in the future clean it up automatically? (which might never happen).

Attached: Memory graph from Admin panel.

mappedByteBufferAllocation.png: (51.61 KiB) Not downloaded yet

If you compare allocated memory here(400MB) with previous one where i used DataInputStream(325MB) There is a 75MB difference in allocated memory...

EDIT:
I restarted the server a few more times to test it, and sometimes it allocates 400MB and other times the old 325MB as with DataInputStream so im now thinking its something else thats happening, but I have no idea what since every time the same thing happens on server startup, there is no randomness or procedural stuff...

Posted: **20 Jan 2025, 07:08**

Garbage collection will happen when the JVM decides it is needed

There's no specific reason to force it, imho.
Is there a specific reason to use a MappedByteBuffer? It seems this is a specialized class used for mapping files in memory... but in your case you just need to load the data and be done with it (if I understood correctly)

Cheers

Posted: **22 Jan 2025, 15:44**

This is my current code, im not sure anymore if it generates extra allocated/used memory though as my findings seem to change with every restart so its inconclusive, but I think you're right, without the MappedByteBuffer it shouldnt use any extra memory, but maybe something internally inside JAVA API is creating or soft refercing the data and keeping it in memory sometimes?

Code: Select all

    public static short[][] LoadNoiseMapFromBinary(String filePath, int size)
    {
        short[][] noiseMap = new short[size][];

        try (FileInputStream fileInputStream = new FileInputStream(filePath);
                FileChannel fileChannel = fileInputStream.getChannel())
        {

            // Map the entire file into memory
            ByteBuffer buffer = fileChannel.map(FileChannel.MapMode.READ_ONLY, 0, fileChannel.size());
            buffer.order(ByteOrder.LITTLE_ENDIAN); // Set byte order to little-endian

            for (int y = 0; y < size; y++)
            {
                noiseMap[y] = new short[size];
                for (int x = 0; x < size; x++)
                {
                    // Read the next short in little-endian order
                    noiseMap[y][x] = buffer.getShort();
                }
            }
        }
        catch (IOException e)
        {
            _ext.trace("Error loading noise maps: " + e.getMessage());
            e.printStackTrace();
        }

        _ext.trace("Loaded noisemap from: " + filePath);
        return noiseMap;
    }
}

Posted: **22 Jan 2025, 18:26**

Hi,

but maybe something internally inside JAVA API is creating or soft refercing the data and keeping it in memory sometimes?

No there is no such mechanism.
If your Extension is holding on some big objects those will be gone by the time the extension is destroyed or earlier if you release them manually (i.e. null-ify all references)

Cheers

forums.smartfoxserver.com

Server loading data from storage very slow

Server loading data from storage very slow

Re: Server loading data from storage very slow

Re: Server loading data from storage very slow

Re: Server loading data from storage very slow

Re: Server loading data from storage very slow

Re: Server loading data from storage very slow

Re: Server loading data from storage very slow

Re: Server loading data from storage very slow

Re: Server loading data from storage very slow

Re: Server loading data from storage very slow