* Refactors and defuckulates dbcore. Adds support for min_threads rustg setting, Reduce query delay, Make unit tests faster (#74852)
dbcore was very fuckulated.
It had 3 lists of queries, but they all had their own current_run style
list to support mc_tick_check (as it was already being done before with
the undeleted query check, so i can understand why they ~~cargo culted~~
mirrored the behavior) This was silly and confusing and unneeded given
two of those loops can only process at most 25 items at a time on
default config, plus these were cheap operations (ask rustg to start
thread, ask rustg to check on thread).
Because of the confusingness of the 6 lists for 3 query states, The code
to run pending/queued queries immediately during world shutdown was
instead looking at the current_run list for active queries, **meaning
those queries got ran twice.**
The queued query system only checked the current active query count in
fire(), meaning even when there was nothing going on in this subsystem
new queries had to wait for the next fire() to run (10 ticks, so 500ms
on default config)
Those have all been fixed.
the config `BSQL_THREAD_LIMIT` has been renamed to
`POOLING_MAX_SQL_CONNECTIONS` and its default was lowered to match
MAX_CONCURRENT_QUERIES .
added a new config `POOLING_MIN_SQL_CONNECTIONS`, allowing you to
pre-allocate a reserve of sql threads.
The queue processing part of SSdbcore's fire() has been made to not obey
mc_tick_check for clarity and to make the following change easier to do:
If there is less than `MAX_CONCURRENT_QUERIES` in the active queue, new
queries activate immediately.
(its ok that there are two configs that kinda do the same thing,
POOLING_MAX_SQL_CONNECTIONS maps to max-threads in the mysql crate, and
it seems to only be a suggestion, meanwhile MAX_CONCURRENT_QUERIES can't
do anything during init, which is when the highest amount of concurrent
queries tend to happen.)
🆑
config: database configs have been updated for better control over the
connection pool
server: BSQL_THREAD_LIMIT has been renamed to
POOLING_MAX_SQL_CONNECTIONS, old configs will whine but still work.
fix: fixed rare race condition that could lead to a sql query being ran
twice during world shutdown.
/🆑
I have not tested this pr.
* Refactors and defuckulates dbcore. Adds support for min_threads rustg setting, Reduce query delay, Make unit tests faster
---------
Co-authored-by: Kyle Spier-Swenson <kyleshome@gmail.com>
* Refactors dbcore and limits the maximum amount of concurrent async queries to a variable amount (#59676)
Refactors dbcore to work off a subsystem if executed async and limits the maximum amount of concurrent async queries to 25.
This has been tested locally on a mysql docker image and there were no crashes (as long as you didn't run it with debug extools) + data was getting recorded fine.
Why It's Good For The Game
May or may not resolve terry crashes, however, each query creates a new thread which takes up 2mb, preventing the game from using that 2mb. This can lead to ooms if they stack up, e.g. due to poor connectivity. This solves that issue.
maintainer note: this did not actually resolve the crashes, but has value anyway. Crashes were sidestepped fixed by finding out Large Address Awareness works
cl
refactor: Refactors dbcore.dm to possibly resolve the crashes that happen on Terry.
/cl
* Fixes an oversight in database code and cleans up telemetry (#64177)
As it is right now, we never actually clear the temporary list processing_queries
So if the subsystem is for some reason unable to complete a run, we will just whip right back around to it again
If it's been long enough, this could even cause horrific log spam. There was just now a manuel round with roughly 30k undeleted query errors. not good.
But what was actually not deleting you may ask?
Well
When you create a db request, a 5 minute timer starts. after those 5 minutes are up, the request is qdeleted by the db subsystem
This is to prevent the creation of unused requests, and to handle requests that are never cleaned up
Telemetry code was creating all of its db requests inside a for loop that could check tick, and then later
attempting to call them in series
Since requests by default sleep, this almost always lead to undeleted queries, which harddel'd given long enough periods
I've fixed this by moving the data gathering away from the query creation
Why is it good for the game
I was working on atmos code, happy, safe in my delusion, when suddenly I got a ping from tattle freaking out over 200 undeleted queries a second
This resolves that issue, so I can once again live in peace
Changelog
cl
admin: Telemetry code will spam you with undeleted query logs much less often now!
server: Improved how the db subsystem handles undeleted queries, should never have an incident like that again
/cl
* Fixes an error in telemetry queries (#64205)
* Hardsynced time_track.dm with upstream
Co-authored-by: Watermelon914 <37270891+Watermelon914@users.noreply.github.com>
Co-authored-by: LemonInTheDark <58055496+LemonInTheDark@users.noreply.github.com>
* Add $reset directive to config processor for setting a config entry to default value
* Update game_options.dm
Co-authored-by: Bobbahbrown <bobbahbrown@gmail.com>
Co-authored-by: Gandalf <jzo123@hotmail.com>
* Refactors dbcore and limits the maximum amount of concurrent async queries to a variable amount (#59676)
Refactors dbcore to work off a subsystem if executed async and limits the maximum amount of concurrent async queries to 25.
This has been tested locally on a mysql docker image and there were no crashes (as long as you didn't run it with debug extools) + data was getting recorded fine.
Why It's Good For The Game
May or may not resolve terry crashes, however, each query creates a new thread which takes up 2mb, preventing the game from using that 2mb. This can lead to ooms if they stack up, e.g. due to poor connectivity. This solves that issue.
maintainer note: this did not actually resolve the crashes, but has value anyway. Crashes were sidestepped fixed by finding out Large Address Awareness works
cl
refactor: Refactors dbcore.dm to possibly resolve the crashes that happen on Terry.
/cl
* Refactors dbcore and limits the maximum amount of concurrent async queries to a variable amount
Co-authored-by: Watermelon914 <37270891+Watermelon914@users.noreply.github.com>
* Add BSQL library v1.1.1.0
* Modify dbcore to use BSQL
* Add missing QDEL_NULL for connectOperation
* Moves BSQL_Shutdown() call to dbcore shutdown
* Fix passing the wrong argument to DBQuery/New()
* Darn it @Jordie0608. Fixes db calls without Connect check
No seriously please make sure I'm not breaking anything
* Queries with a null connection won't runtime
* Fix quoting
* Fix mistake
* Update BSQL to v1.1.2.0
* Update BSQL DMAPI to v1.0.1.0
* Fix connection instatiation
* Does the smart thing in regards to quoting
* Fix braces
* Update BSQL to 1.2.0.0. DMAPI to 1.1.0.0
* Execute/NextRow/MassInsert now have async parameter
* Build BSQL for tests
* Add missing apt source
* Def still need gcc-multilib
* Wut
* Revert "Wut"
This reverts commit d7c98a9a6b27f6db03e9f5cc534650d59d018048.
* Try this then
* Could it really be that simple?
* Literally running out of ideas here
* Update BSQL to v1.2.1.0 DMAPI to v1.1.1.0
* Update BSQL travis version
* Nothing about this makes sense tbqhwyfam
* Whoo boy
* No idea why this isn't working tbh
* Absolute madness
* Ahhhhhhhhhhhhh
* *deep breath*
* "though yet again i was frustrated by failure"
* Add BSQL to Dockerfile
* Pass through MassInsert async param
* BSQL to v1.3.0.0 DMAPI to 1.2.0.0
* Add timeout support
* Wait, something's fucky
* Wtf is this meme?
* Just get good lmao
* Just stop being shit lol
* Stupid verbose logging
* Remove verbosity
* Good god
* BSQL to v1.3.0.1 DMAPI to v1.2.0.1
* BSQL to v1.3.0.2
* Update BSQL travis version
* Update BSQL docker version
* Didn't mean to change that
* Strip connection information from debug logs and make it configgable
* Move this to where CONFIG_GET is defined
This works around some vv-fu you can use to pass in a config_entry to a proc that reads the value var of some other datum. Byond is stupid enough to actually read it, so it must be uniquely named.
* Configuration datum refactor
* More WIP
* New easier on the eyes format
* More WIP
* Finished config.txt
* Fucktons more WIP
* The end of conversion draws near...
* Add all this shit
* Done converting entries finally
* Hunting down compile errors
* More WIP
* MORE CONVERSIONS
* More WIP
* More WIP
* Oh shit only 90 errors this time!
* IT COMPILES!!!
* Fixes world start runtimes
