Speedup Filesearch #197

2025-08-10T20:36:01+02:00

alufolie91 commented

2025-08-10 20:36:01 +02:00

(Migrated from github.com)

Currently the game checks almost ALL folders in no particular order for addons, this includes the log folder, replay folder etc., which shouldnt ever contain addons.
This implements a folder "blacklist" of sorts, entirely skipping folders like these, and as a result, skipping checking potentially multiple thousand files for every single addon.
Ontop of this, after startup, findfile will now check the download, addons and custom addon folder first in line, since those are where addons reside by default

Windows also now uses the platform specific "GetFileAttributes" instead of stat (which is very slow as it seemingly has to emulate some POSIX behaviour), both for filesearch and preparefilemenu
Currently this ignores symlinks, need some more looking into if there is a need to follow those on windows

This MASSIVELY speeds up seeking for addons on Windows especially
even more if you got many replays, logfiles screenshots and such

Need a bit of testing to make sure this works as intended

Comparison before

https://github.com/user-attachments/assets/2923132b-aa13-4a12-9bcd-cf9c73be4e88

after:

https://github.com/user-attachments/assets/a0ca4d77-61cf-4cf5-a0ce-c5d68c88603a

(empty download/addons folder, many replays and logs)

Many thanks to Sunflower for testing this out!

Currently the game checks almost ALL folders in no particular order for addons, this includes the log folder, replay folder etc., which _shouldnt_ ever contain addons. This implements a folder "blacklist" of sorts, entirely skipping folders like these, and as a result, skipping checking potentially multiple thousand files for every single addon. Ontop of this, after startup, findfile will now check the download, addons and custom addon folder first in line, since those are where addons reside by default Windows also now uses the platform specific "GetFileAttributes" instead of stat (which is very slow as it seemingly has to emulate some POSIX behaviour), both for filesearch and preparefilemenu Currently this ignores symlinks, need some more looking into if there is a need to follow those on windows This MASSIVELY speeds up seeking for addons on Windows especially even more if you got many replays, logfiles screenshots and such Need a bit of testing to make sure this works as intended Comparison before https://github.com/user-attachments/assets/2923132b-aa13-4a12-9bcd-cf9c73be4e88 after: https://github.com/user-attachments/assets/a0ca4d77-61cf-4cf5-a0ce-c5d68c88603a (empty download/addons folder, many replays and logs) Many thanks to Sunflower for testing this out!

Indev450 (Migrated from github.com) approved these changes

2025-08-12 16:56:23 +02:00

alufolie91 commented

2025-08-27 10:57:03 +02:00

(Migrated from github.com)

Did some more changes for this
fixed some portability issues for platforms besides linux/freebsd and windows
also changed this to actually only check the folder blacklist if the folder actually changed during file traversal, so we can save potentially multiple thousands of string comparisons, this speeds up the process even further superseeded by https://github.com/Indev450/SRB2Kart-Saturn/pull/197/commits/ef9472097d019bfd82ad32fed896031e67c99bd8 much thanks to toaster for revisiting my initial implementation of the folder blacklist system
Only thing im still unsure about if we should support symlinks on Windows builds, from experience, symlinks arent used much on windows systems, best would be to wait for some public testing on this

Did some more changes for this fixed some portability issues for platforms besides linux/freebsd and windows ~~also changed this to actually only check the folder blacklist if the folder actually changed during file traversal, so we can save potentially multiple thousands of string comparisons, this speeds up the process even further~~ superseeded by https://github.com/Indev450/SRB2Kart-Saturn/pull/197/commits/ef9472097d019bfd82ad32fed896031e67c99bd8 much thanks to toaster for revisiting my initial implementation of the folder blacklist system Only thing im still unsure about if we should support symlinks on Windows builds, from experience, symlinks arent used much on windows systems, best would be to wait for some public testing on this

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: saturn/SRB2Kart-Saturn#197