1 (edited by frothypants 2022-05-09 18:46:21)

Topic: Major performance issues

Hey there, using the source version with Photon PUN and am starting to run into some major performance issues now that our character count is growing.

We have around 40 3D fighters (loaded using the resource method) and 4 stages(both will continue to grow) but it looks like all characters are in memory, even the unused ones since it's all just one scene. We're using the 3d model character select screen instead of a flat 2D version.

On the loading battle screen, I'm doing this to try and free things up in case referencing them on char select screen is causing them to be stored in memory but it doesn't seem to help. All characters are still referenced in the

        GC.Collect();
        Resources.UnloadUnusedAssets();

Are there any suggestions on how to best manage a large number of characters and levels so that they aren't loaded into memory unless needed and are unloaded once they are no longer needed?

Continuing to look at this, in the profiler if I do a player v player game and just let it sit there without any input the GC memory allocation steadily climbs. It looks like a lot of that is coming from UFE.FixedUpdate() and at a quick glance it looks like there are a lot of variables declared in the loop in every single fixed update? Could that be part of the problem?

Thanks!

Share

Thumbs up Thumbs down

Re: Major performance issues

As far as the resources system goes I don't know much about it but, I'm going to avoid using it for now.
Take a look at this: https://www.reddit.com/r/Unity3D/commen … ameobject/
I wouldn't take the info in there as fact but it's something to think about.

https://i.imgur.com/VZ5pd0i.png

As far as GC goes from UFE, with how complicated UFE is it's pretty much bound to generate garbage.
Without knowing UFE's complex code it's tough to pin down what exactly is causing it.
You can deep profile, but ultimately it's probably gonna be on @MisterMind to try and reduce GC.

Share

Thumbs up +1 Thumbs down

3 (edited by frothypants 2022-05-10 07:51:38)

Re: Major performance issues

Thanks for getting bcak to me and the link. I agree that asset bundles / Addressables is going to be the way for us long term. Since I bought the source version I plan to eventually refactor the character prefab loading system to use Addressables instead to reduce memory and also because we're building for WebGL and are already hitting the ceiling there for how much we can have loaded into memory at runtime.

Seems like most of the garbage collection is happening from the use of LINQ in various places. For example, from FluxStateTracker.cs beginning at line 559

                // Inputs being held down (charges)
                controlsScript.inputHeldDown = state.inputHeldDown.ToDictionary(entry => entry.Key, entry => entry.Value);
                

                // Projectiles
                controlsScript.projectiles = new List<ProjectileMoveScript>();
                foreach (ProjectileMoveScript projectile in state.projectiles) {
                    if (projectile != null) controlsScript.projectiles.Add(projectile);
                }

                
                // Buttons Pressed
                controlsScript.MoveSet.lastButtonPresses = new List<ButtonSequenceRecord>();
                foreach (ButtonSequenceRecord btnRecord in state.moveSet.lastButtonPresses) {
                    controlsScript.MoveSet.lastButtonPresses.Add(new ButtonSequenceRecord(btnRecord.buttonPresses, btnRecord.chargeTime));
                }

                // Cooldown Timers
                controlsScript.MoveSet.lastMovesPlayed = state.moveSet.lastMovesPlayed.ToDictionary(entry => entry.Key, entry => entry.Value);

ToDictionary() is the 2nd worst method in terms of garbage collection which is probably why it's steadily climbing there.
https://www.jacksondunstan.com/articles/4840

Because getting UFE 2 to a production ready state for us is going to require a bunch of refactoring and UFE 3 is on the horizon I wonder if @Mistermind would be willing to work something out with us. I'd gladly preorder the source version and contribute to the codebase. I'm paying for the top tier patreon right now to get access to the repo but it seems that the UFE 2 repo has been abandoned in favor of UFE 3 development.

Edit: a few other issues we've run into that I plan to dig in and fix

* If two players begin a search for a match at the same time, neither will find a match and both will set up a room. There's no timeout to go back and search again so they'll both just sit in their room indefinitely thinking no one else is online to play

* Randomly get a bunch of "end of stream" errors in the console from PUN. haven't been able to troubleshoot and dig in yet.

* The one scene and instantiation flow seems to be causing quite a few issues. Plan to replace this with scenes and singletons for anything that needs to be persistent.

* Several other small things we plan to fix. Would much rather put this energy towards UFE 3 though.

Share

Thumbs up Thumbs down

Re: Major performance issues

Is there a chance you could switch to prefab loading instead of resources? Have you tried that method before?

Yea LINQ can be pretty bad for GC, I was using .ToList() temporarily for something but quickly found a solution to not need it anymore.

Turns out, .ToList() creates a new list which generate garbage? I haven't profiled it yet but, from what i've read that's probably the case.

Finding ways to avoid LINQ would help for sure.

I think I've encountered some of those PUN issues you've ran into.

Share

Thumbs up Thumbs down

Re: Major performance issues

I was originally using the prefab route but when we started to run into memory resource issues in WebGL I switched to the resources method based on the recommendation of the final bullet in the Notes section here:

http://www.ufe3d.com/doku.php/character:prefabs

I haven't dug into the flux scripts enough to understand what's going on in the places that LINQ is used but will need to do that it seems. In a fighting game, with all the rapid mashing of buttons that LINQ caused GC in every frame is probably where the steady GC allocated memory issues are coming from.

Another issue that I think is going on is the lack of object pooling. There's some kind of preloading / memory allocation happening for the projectiles and hit effects that I think is also attributing to the problem. Will need to also rework that likely in order to instantiate a pool and recycle for all of the particle/hit effects.

With all of this work I'll be doing, I'd much rather invest this energy in UFE 3 if that were possible. Happy to contribute.

Share

Thumbs up Thumbs down

Re: Major performance issues

This has some good info in it: http://www.ufe3d.com/forum/viewtopic.php?id=2709

Share

Thumbs up Thumbs down

Re: Major performance issues

Looks like three other devs went down the same path that I have and none of them pursued a solution. Might try disabling the netcode rollback and see if that helps at all.

The input dictionary declaration breaking the input system doesn't sound like good times. We're using rewired for controller support so that will probably add a layer of difficulty to the improvements.

Between these performance issues, now likely needing to get rid of the netcode rollback, and having to abandon Fuzzy AI for WebGL we've had to make a lot of compromises for a $500 framework. Really hoping we can just get our hands on an early version of UFE 3 rather than pouring more energy into this version that is being moved away from.

Share

Thumbs up Thumbs down

Re: Major performance issues

UFE 2 might still be supported even after UFE 3 is released.
It won't be easy but, is probably possible to reduce GC
We'll see in time.

Share

Thumbs up Thumbs down

Re: Major performance issues

Possibly but the UFE 2 repo I'm paying for access to on Patreon hasn't been updated in 8-9 months.

We're going live with a public beta at the end of the month so time isn't on our side. Hopefully @Mistermind pops in sometime soon and provides some input.

Share

Thumbs up +1 Thumbs down

Re: Major performance issues

There are a few reasons why this could be happening.
Are you preloading your stages and move sets (loading method - Resource)?
These systems were made to take advantage of the resource folder provided by Unity. For large rosters it is highly recommended you use these loading methods.

If you are, it sounds like the issues are being caused by unknown prefabs or scripts you are using (specially if they are singletons), which can linger in the memory, causing issues to the GC.

In the case of UFE 3, the main different is that the engine will now dedicate an entire scene for the game mechanics only. The scene file will act as the stage instead of a prefab, and every UFE event listener will be automatically deleted once the scene is unloaded.

If the problem is related to the "single scene issue" and you can't narrow down the cause, there is a work around, but you need to build your own menu system. UFE 2 provides a somewhat "easy to use" user interface out of the box, but you don't necessarily have to use it. In fact I recommend advanced users not to use the option "Full Interface", but instead "Versus Mode" under Deployment Options. You can then load UFE in a separated scene and programmatically load the selected characters in. For more on this, check out this link:
http://www.ufe3d.com/doku.php/code

Like UFE? Please rate and review us on the Asset Store!
Questions about the Forum? Check out our Karma FAQ.
Don't forget to check our discord channel.

11 (edited by frothypants 2022-05-13 15:05:10)

Re: Major performance issues

Mistermind wrote:

There are a few reasons why this could be happening.

If you are, it sounds like the issues are being caused by unknown prefabs or scripts you are using (specially if they are singletons), which can linger in the memory, causing issues to the GC.


If the problem is related to the "single scene issue" and you can't narrow down the cause, there is a work around, but you need to build your own menu system. UFE 2 provides a somewhat "easy to use" user interface out of the box, but you don't necessarily have to use it. In fact I recommend advanced users not to use the option "Full Interface", but instead "Versus Mode" under Deployment Options.

I'm using the resource load method for both the characters and the stages.

No custom scripts or prefabs unless a stage that is loaded by resources that contains prefabs within it is an issue.

Based on my profiling and other's experience here in the forums it seems like most of the GC is coming from the input system, flux classes, rollback netcode. After disabling the rollback netcode, users have reported fewer disconnects but then timing of blocks and counters is a mess since host has the advantage then. Used the map recorder for all characters but didn't make a difference with disconnects.

Profiler shows about 170mb-230mb of GC used memory depending on if preloading hit effects/stage are enabled. Warm all shaders crashes play mode every time so that's been off for a while.

edit: Tested the versus only deployment method and no difference to the total memory or GC memory used.

Share

Thumbs up Thumbs down

Re: Major performance issues

frothypants wrote:
Mistermind wrote:

There are a few reasons why this could be happening.

If you are, it sounds like the issues are being caused by unknown prefabs or scripts you are using (specially if they are singletons), which can linger in the memory, causing issues to the GC.


If the problem is related to the "single scene issue" and you can't narrow down the cause, there is a work around, but you need to build your own menu system. UFE 2 provides a somewhat "easy to use" user interface out of the box, but you don't necessarily have to use it. In fact I recommend advanced users not to use the option "Full Interface", but instead "Versus Mode" under Deployment Options.

I'm using the resource load method for both the characters and the stages.

No custom scripts or prefabs unless a stage that is loaded by resources that contains prefabs within it is an issue.

Based on my profiling and other's experience here in the forums it seems like most of the GC is coming from the input system, flux classes, rollback netcode. After disabling the rollback netcode, users have reported fewer disconnects but then timing of blocks and counters is a mess since host has the advantage then. Used the map recorder for all characters but didn't make a difference with disconnects.

Profiler shows about 170mb-230mb of GC used memory depending on if preloading hit effects/stage are enabled. Warm all shaders crashes play mode every time so that's been off for a while.

edit: Tested the versus only deployment method and no difference to the total memory or GC memory used.

The profiler can sometimes give false positives, but this could be a cumulative issue caused by a large input buffer (even if its offline).
Under Network Options -> Rollback Netcode, what is the value for your Input Buffer Size?

Like UFE? Please rate and review us on the Asset Store!
Questions about the Forum? Check out our Karma FAQ.
Don't forget to check our discord channel.

Re: Major performance issues

Mistermind wrote:

The profiler can sometimes give false positives, but this could be a cumulative issue caused by a large input buffer (even if its offline).
Under Network Options -> Rollback Netcode, what is the value for your Input Buffer Size?

True, but not with the progressively growing GC allocated memory every second. That increase is consistent with users reporting that the disconnects happen near the end of the match every time, when the garbage has been buliding throughout the match from the input, particle, and effect instantiation.

Watching the GC allocated memory on the profiler shows it steadily growing by several MB per minute. We're building for WebGL where memory is very limited so this consistent increase in memory is likely what is causing the end-of-match disconnects.

Tried using the default network options originally, this is what I was using at one point, then disabled netcode altogether which seemed to help somewhat.
network

Share

Thumbs up Thumbs down