## About The Pull Request
When I optimized maploading in (#69632d34fa4c642) I made some assumptions based
on the TGM format, so how newlines worked and such.
This would break attempts to load maps in the dmm format, so maps edited
in dreammaker and not commited yet.
I've fixed this by adding a map format detection system This has allowed
me to further optimize tgm by taking advantage of a list of assumptions.
I printed them out in a comment near the top of the file.
This showed the most in build_cache, which went from 0.7 seconds to 0.4
on my machine.
The optimizations here involve using regex to pull out var edits,
parsing models line by line rather then all at once, which allows for
much cheaper parsing, and avoiding a safety trim()
I did also save like 0.05 seconds in load, because I could pull a bunch
of operations out to once per map, because of prior knowledge of where
the max/min x and y's would be.
## Why It's Good For The Game
Speed. Also ensures mappers don't get fucked over.
Co-authored-by: Mothblocks <35135081+Mothblocks@users.noreply.github.com>
## About The Pull Request
Area contents isn't a real list, instead it involves filtering
everything in world
This is slow, and something we should have better support for.
So instead, lets manage a list of turfs inside our area. This is simple,
since we already move turfs by area contents anyway
This should speed up the uses I've found, and opens us up to using this
pattern more often, which should make dev work easier.
By nature this is a tad fragile, so I've added a unit test to double
check my work
Rather then instantly removing turfs from the contained_turfs list, we
enter them into a list of turfs to pull out, later.
Then we just use a getter for contained_turfs rather then a var read
This means we don't need to generate a lot of usage off removing turf by
turf from space, and can instead do it only when we need to
I've added a subsystem to manage this process as well, to ensure we
don't get any out of memory errors. It goes entry by entry, ensuring we
get no overtime.
This allows me to keep things like space clean, while keeping high
amounts of usage on a sepearate subsystem when convienient
As a part of this goal of keeping space's churn as low as possible, I've
setup code to ensure we do not add turfs to areas during a z level
increment adjacent mapload. this saves a LOT of time, but is a tad
messy
I've expanded where we use contained_turfs, including into some cases
that filter for objects in areas. need to see if this is sane or not.
Builds sortedAreas on demand, caching until we mark the cache as
violated
It's faster, and it also has the same behavior
I'm not posting speed changes cause frankly they're gonna be a bit
scattered and I'm scared to.
@Mothblocks if you'd like I can look into it. I think it'll pay for
itself just off `reg_in_areas_in_z` (I looked into it. it's really hard
to tell, sometimes it's a bit slower (0.7), sometimes it's 2 seconds
(0.5 if you use the old master figure) faster. life is pain.)
## Why It's Good For The Game
Less stupid, more flexible, more speed
Co-authored-by: san7890 <the@san7890.com>
* Makes only station areas part of Statioj
* Makes only subtypes of /area/station be part of the station
* Removes Icemoon and Shuttles as a check for Anomaly placers as they aren't needed anymore, not being part of shuttles.
* Removes a ton of uses of NO_ALERTS where it is no longer needed.
Did you know that areas can be non unique (unique per area)
I didn't, and I wrote mapping code such that we'd make all areas unique
This fixes that. It costs 0.1 seconds but without it areas don't work
anymore
* 'optimizes' space transitions by like 0.06 seconds, makes them easier to read tho, so that's an upside
* ''''optimizes'''' parsed map loading
I'm honestly not sure how big a difference this makes, looked like small
percentage points if anything
It's a bit more internally concistent at least, which is nice. Also I
understand the system now.
I'd like to think it helped but I think this is kinda a "do you think
it's easier to read" sort of situation. if it did help it was by the
skin of its teeth
* Saves 0.6 seconds off loading meta and lavaland's map files
This is just a lot of micro stuff.
1: Bound checks don't need to be inside for loops, we can instead bound the iteration counts
2: TGM and DMM are parsed differently. in dmm a grid_set is one z level,
in tgm it's one collumn. Realizing this allows you to skip copytexts and
other such silly in the tgm implemenentation, saving a good bit of time
3: Min/max bounds do not need to be checked inside for loops, and can
instead be handled outside of them, because we know the order of x
and y iteration. This saves 0.2 seconds
I may or may not have made the code harder to read, if so let me know
and I'll check it over.
* Micro ops key caching significantly. Fixes macros bug
inserting \ into a dmm with no valid target would just less then loop
the string. Dumb
Anyway, optimizations. I save a LOT of time by not needing to call
find_next_delimiter_position for every entry and var set. (like maybe 0.5
seconds, not totally sure)
I save this by using splittext, which is significantly faster. this
would cause parsing issues if you could embed \n into dmms, but you
can't, so I'm safe.
Lemme see uh, lots of little things, stuff that's suboptimal or could be
done cheaper. Some "hey you and I both know a \" is 2 chars long sort of
stuff
I removed trim_text because the quote trimming was never actually used,
and the space trimming was slower then using the code in trim. I also
micro'd trim to save a bit of time. this saves another maybe 0.5.
Few other things, I think that's the main of it. Gives me the fuzzy
feelings
* Saves 50% of build_coordinate's time
Micro optimizing go brrrrr
I made turf_blacklist an assoc list rather then just a normal one, so
lookups are O(log n) instead of O(n). Also it's faster for the base case
of loading mostly space.
Instead of toggling the map loader right before and right after New()
calls, we toggle at the start of mapload, and disable then reenable if
we check tick. This saves like 0.3 seconds
Rather then tracking an area cache ourselves, and needing to pass it
around, we use a locally static list to reference the global list of
area -> type. This is much faster, if slightly fragile.
Rather then checking for a null turf at every line, we do it at the
start of the proc and not after. Faster this way, tho it can in theory
drop area vvs.
Avoids calling world.preloader_setup unless we actually have a unique
set of attributes. We use another static list to make this comparison
cheap. This saves another 0.3
Rather then checking for area paths in the turf logic, or vis versa, we
assume we are creating the type implied by the index we're reading off.
So only the last type entry will be loaded like a turf, etc.
This is slightly unsafe but saves a good bit of time, and will properly
error on fucked maps.
Also, rather then using a datum to hold preloader vars, we use 2 global
variables. This is faster.
This marks the end of my optimizations for direct maploading. I've
reduced the cost of loading a map by more then 50% now. Get owned.
* Adds a define for maploading tick check
* makes shuttles load again, removes some of the hard limits I had on the reader for profiling
* Macro ops cave generation
Cave generation was insanely more expensive then it had any right to be.
Maybe 0.5 seconds was saved off not doing a range(12) for EVERY SPAWNED
MOB.
0.14 was saved off using expanded weighted lists (A new idea of mine)
This is useful because I can take a weighted list, and condense it into
weight * path count. This is more memory heavy, and costs more to
create, but is so much faster then the proc.
I also added a naive implementation of gcd to make this a bit less bad.
It's not great, but it'll do for this usecase.
Oh and I changed some ChangeTurfs into New()s. I'm still not entirely
sure what the core difference between the two is, but it seems to work
fine.
I believe it's safe because the turf below us hasn't init'd yet, there's
nothing to take from them. It's like 3 seconds faster too so I'll be sad
when it turns out I'm being dumb
* Micros river spawning
This uses the same sort of concepts as the last change, mostly New being
preferable to ChangeTurf at this level of code.
This bit isn't nearly as detailed as the last few, I honestly got a bit
tired. It's still like 0.4 seconds saved tho
* Micros ruin loading
Turns out it saves time if you don't check area type for every tile on a
ruin. Not a whole ton faster, like 0.03, but faster.
Saves even more time (0.1) to not iterate all your ruin's turfs 3 times
to clear away lavaland mobs, when you're IN SPACE who wrote this.
Oh it also saves time to only pull your turf list once, rather then 3
times
a month or two ago i realized that on master the reason why get_hearers_in_view() overtimes so much (ie one of our highest overtiming procs at highpop) is because when you transmit a radio signal over the common channel, it can take ~20 MILLISECONDS, which isnt good when 1. player verbs and commands usually execute after SendMaps processes for that tick, meaning they can execute AFTER the tick was supposed to start if master is overloaded and theres a lot of maptick 2. each of our server ticks are only 50 ms, so i started on optimizing this.
the main optimization was SSspatial_grid which allows searching through 15x15 spatial_grid_cell datums (one set for each z level) far faster than iterating over movables in view() to look for what you want. now all hearing sensitive movables in the 5x5 areas associated with each spatial_grid_cell datum are stored in the datum (so are client mobs). when you search for one of the stored "types" (hearable or client mob) in a radius around a center, it just needs to
iterate over the cell datums in range
add the content type you want from the datums to a list
subtract contents that arent in range, then contents not in line of sight
return the list
from benchmarks, this makes short range searches like what is used with radio code (it goes over every radio connected to a radio channel that can hear the signal then calls get_hearers_in_view() to search in the radios canhear_range which is at most 3) about 3-10 times faster depending on workload. the line of sight algorithm scales well with range but not very well if it has to check LOS to > 100 objects, which seems incredibly rare for this workload, the largest range any radio in the game searches through is only 3 tiles
the second optimization is to enforce complex setter vars for radios that removes them from the global radio list if they couldnt actually receive any radio transmissions from a given frequency in the first place.
the third optimization i did was massively reduce the number of hearables on the station by making hologram projectors not hear if dont have an active call/anything that would make them need hearing. so one of hte most common non player hearables that require view iteration to find is crossed out.
also implements a variation of an idea oranges had on how to speed up get_hearers_in_view() now that ive realized that view() cant be replicated by a raycasting algorithm. it distributes pregenerated abstract /mob/oranges_ear instances to all hearables in range such that theres at max one per turf and then iterates through only those mobs to take advantage of type-specific view() optimizations and just adds up the references in each one to create the list of hearing atoms, then puts the oranges_ear mobs back into nullspace. this is about 2x as fast as the get_hearers_in_view() on master
holy FUCK its fast. like really fucking fast. the only costly part of the radio transmission pipeline i dont touch is mob/living/Hear() which takes ~100 microseconds on live but searching through every radio in the world with get_hearers_in_radio_ranges() -> get_hearers_in_view() is much faster, as well as the filtering radios step
the spatial grid searching proc is about 36 microseconds/call at 10 range and 16 microseconds at 3 range in the captains office (relatively many hearables in view), the new get_hearers_in_view() was 4.16 times faster than get_hearers_in_view_old() at 10 range and 4.59 times faster at 3 range
SSspatial_grid could be used for a lot more things other than just radio and say code, i just didnt implement it. for example since the cells are datums you could get all cells in a radius then register for new objects entering them then activate when a player enters your radius. this is something that would require either very expensive view() calls or iterating over every player in the global list and calling get_dist() on them which isnt that expensive but is still worse than it needs to be
on normal get_hearers_in_view cost the new version that uses /mob/oranges_ear instances is about 2x faster than the old version, especially since the number of hearing sensitive movables has been brought down dramatically.
with get_hearers_in_view_oranges_ear() being the benchmark proc that implements this system and get_hearers_in_view() being a slightly optimized version of the version we have on master, get_hearers_in_view_as() being a more optimized version of the one we have on master, and get_hearers_in_LOS() being the raycasting version currently only used for radios because it cant replicate view()'s behavior perfectly.
Done using this command sed -Ei 's/(\s*\S+)\s*\t+/\1 /g' code/**/*.dm
We have countless examples in the codebase with this style gone wrong, and defines and such being on hideously different levels of indentation. Fixing this to keep the alignment involves tainting the blames of code your PR doesn't need to be touching at all. And ultimately, it's hideous.
There are some files that this sed makes uglier. I can fix these when they are pointed out, but I believe this is ultimately for the greater good of readability. I'm more concerned with if any strings relied on this.
Hi codeowners!
Co-authored-by: Jared-Fogle <35135081+Jared-Fogle@users.noreply.github.com>
Refactors the holodeck to use map templates instead of copy_contents_to, which every maintainer seems to have complaints about.
Fixes#41485 because the matches become part of the spawned list created by ssatoms
Fixes#54789 because the holodeck area no longer has the NO_TELEPORT flag
Fixes#55676 because the map templates cant be changed midround unlike the program copies in the centcom z level
Fixes#49318 because the holodeck no longer creates new areas like the original did
This pr also changes initTemplateBounds to be a /datum/map_template proc instead of a parsed_map proc. This was mainly so I wouldn't have to duplicate vars between map_template and parsed_map. It's also nice because there's no longer a parsed_map proc inside the map_template file, especially when it didn't need to be a parsed_map proc.
The holodeck sims wont take up space in the centcom z level any more (which allows for more possible programs in the future), and map templates are more heavily tested. This is also a chance to future proof the holodeck against bugs. Holodeck also seems more responsive. This should allow for a second custom holodeck in some future ruin as well, although that of course will not be in play for the near future because of the offstation content ban. Also I documented the fuck out of the holodeck
- Backtick-escape code samples which contain `[]` syntax.
- Fix all crosslinks to nonexistent symbols.
- Somewhat improve docs for qdel defines, research defines, dynamic mode, and others.
- Remove unused bloodcrawling defines.
Some crosslinks to defined but undocumented symbols remain. For BYOND builtins, a future dmdoc version may link those symbols to their entries in the DM reference. Other symbols could be documented by a future PR.
New "file" crosslinks as used in `research.dm` are slated for release in a future dmdoc version.
* Build SpacemanDMM from source
* oops
* Fix new unreachables/dumb code
* ooops cache conflict
* bugfix
* oops
* lint
* ninjanomnom held me down and forced me to delete this
* Unicode support Part 2 -- copytext()
This is the transition of all copytext() calls to be unicode aware and also some nearby calls in the same functions. Most things are just replacing copytext() with copytext_char() as a terrible character limiter but a few others were slightly more involved.
I replaced a ton of
````
var/something = sanitize(input())
something = copytext(something, 1, MAX_MESSAGE_LEN)
````
with a single stripped_input() call. stripped_input() already calls html_encode(), trim(), and some other sanitization so there shouldn't be any major issues there.
This is still VERY rough btw; DNA is a mess, the status displays are complete ass, there's a copytext() in code\datums\shuttles.dm that I'm not sure what to do with, and I didn't touch anything in the tools folder. I haven't tested this much at all yet, I only got it to compile earlier this morning. There's also likely to be weird bugs until I get around to fixing length(), findtext(), and the rest of the string procs.
* Makes the code functional
* Assume color hex strings are always # followed by ascii.
Properly encodes and decodes the stuff in mob_helpers.dm which fixes some issues there.
* Removes ninjaspeak since it's unused
Wasn't setting type to load into before.
Technically this still doesn't mirror inbuilt loader 1:1 because different
mapedits should create separate instances but let's leave that for now.
This was resulting in for example loaded areas with dynamic lighting to 0 to
not initialize properly.
* Move the preloader datum to its own file
* Prettify some of the map loader docs
* Use src rather than usr in map template verbs
* Cache parsed templates between upload and first use
* Validate map templates uploaded by admins before use
* Add href token to validation report links
* Remove duplicate initTemplateBounds method
* Move dmmRegex var to the type that actually uses it
* Keep a single, static map_loader datum
* Combine maploader into parsed_map
* Remove some redundant parameters from parsed_map/load
* Move file2text into parsed_map/New
* Narrow the scope of the map loading lock
* Remove redundant xcrdStart var
* Fix argument names in station reloader
* Move offsets from parsing to loading only
* Tidy argument handling on parsed_map/load
* Store modelCache, track nonexistent atom paths
* Use raw strings to specify the regexes
* Remove cropMap from parsed_map/New
* Don't parse maps twice when checking multi-z validity
Speeds up SSmapping init by approximately 10%.
* Bring the documentation (somewhat) up to speed
* No, I don't know why this fixes it
* Remove ZLEVEL_STATION_PRIMARY
* Add Up and Down traits for use by ladders and chasms
* Give map_config creation its own proc
* Combine LoadConfig and ValidateJSON and remove transition_config
* Make space linkage a z-level trait
* Remove ZLEVEL_EMPTY_SPACE
* Update uses of GetFullMapPath
* Handle multi-Z stations and load Lavaland and Reebe at runtime
* Remove unused space maps
* Fix inappropriate z-expansion in map reader, improve logging
* Update comments relating to z-level configuration
* Add Lavaland and Reebe to ALL_MAPS
* Revert "all this wrapping and it's not even christmas (#33035)"
This reverts commit faaf151580.
* Revert "fuck me for forgetting to graph this one"
This reverts commit 45d7acea2f.
* Revert "defines math"
This reverts commit 2817a1737b.
Badmins: Buildmode map generators have names in the list to select them, instead of paths.
Also, a new map generator has been added, repair/reload station. Use it VERY sparingly, it deletes the block of the map and reloads it to roundstart. THIS CAN CAUSE ISSUES WITH MACHINES AND ATMOSPHERICS, SO DO NOT USE IT UNLESS YOU ABSOLUTELY HAVE TO!
The reload station one tagged DO NOT USE shouldn't be used as it doesn't delete anything before loading, so if you use it you'll have two copies of things. That can result in a LOT of issues, so don't use it unless you're a codermin and know what you're doing/abusing!
* Add the system for managed global variables
* Travis ban old globals
* So you CAN inline proccall, that's neat
* Fix that
* master.dm
* Remove the hack procs
* Move InitGlobals to the proper spot
* configuration.dm
* Fix the missing pre-slash
* clockcult.dm
* This is probably for the best
* Doy
* Fix shit
* Rest of the DEFINES tree
* Fix
* Use global. for access
* Update find_references_in_globals
Always hated that proc
Whoever made it must've bee a r e a l idiot...
* __HELPERS tree
* Move global initialization to master.
Fix the declaration
* database.dm
* Dat newline
* I said DECLARATIVE order!
* Here's something you can chew on @Iamgoofball
* game_modes.dm
* Fix this
* genetics.dm
* flavor_misc.dm
* More stuff
* Do it mso's way. Keep the controllers as global
* Make master actually see it
* Fix
* Finish _globalvars/lists
* Finish the rest of the _globalvars tree
* This is weird
* Migrate the controllers
* SLOTH -> GLOB
* Lighting globals
* round_start_time -> ticker
* PAI card list -> pai SS
* record_id_num -> static
* Diseases list -> SSdisease
* More disease globals to the SS
* More disease stuff
* Emote list
* Better and better
* Bluh
* So much stuff
* Ahh
* Wires
* dview
* station_areas
* Teleportlocs
* blood_splatter_icons
* Stuff and such
* More stuff
* RAD IO
* More stuff and such
* Blob shit
* Changeling stuff
* Add "Balance" to changelogs
* Balance for changelog compiler + Auto Tagging
* Update the PR template
* hivemind_bank
* Bip
* sacrificed
* Good shit
* Better define
* More cult shit
* Devil shit
* Gang shit
* > borers
Fix shit
* Rename the define
* Nuke
* Objectives
* Sandbox
* Multiverse sword
* Announce systems
* Stuff and such
* TC con
* Airlock
* doppllllerrrrrr
* holopads
* Shut up byond you inconsistent fuck
* Sneaky fuck
* Burp
* Bip
* Fixnshit
* Port without regard
* askdlfjs;
* asdfjasoidojfi
* Protected globals and more
* SO MANY
* ajsimkvahsaoisd
* akfdsiaopwimfeoiwafaw
* gsdfigjosidjfgiosdg
* AHHHHHHHHHHHHHHHHHHHHHHH!!!!!
* facerolll
* ASDFASDFASDF
* Removes the unused parts of dmm_suite
* WIP
* Fix quote
* asdfjauwfnkjs
* afwlunhskjfda
* asfjlaiwuefhaf
* SO CLOSE
* wwwweeeeeewwwww
* agdgmoewranwg
* HOLY MOTHER OF FUCK AND THATS JUST HALF THE JOB?!?
* Fix syntax errors
* 100 errors
* Another 100
* So many...
* Ugh
* More shit
* kilme
* Stuuuuuufffff
* ajrgmrlshio;djfa;sdkl
* jkbhkhjbmjvjmh
* soi soi soi
* butt
* TODAY WE LEARNED THAT GLOBAL AND STATIC ARE THE EXACT SAME FUCKING THING
* lllllllllllllllllllllllllllllllllllllllllll
* afsdijfiawhnflnjhnwsdfs
* yugykihlugk,kj
* time to go
* STUFFF!!!
* AAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHH!!!!!!!!!!!!!!!!!!!!!!!
* ngoaijdjlfkamsdlkf
* Break time
* aufjsdklfalsjfi
* CONTROL KAY AND PRAY
* IT COMPILEELEELELAKLJFKLDAFJLKFDJLADKJHFLJKAJGAHIEJALDFJ!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
* Goteem
* Fix testing mode
* This does not belong in this PR
* Convert it to a controller
* Eh, fuck this option
* Revert controllerization Ill do it some other time
* Fix
* Working controllerization
* FOR THE LOVE OF CHRIST PROTECT THE LOGS
* Protect admins and deadmins
* Use the inbuilt proc
* Cut out an unecessary locate call
* Remove an unecessary var
* Add an appropriate warning
* Move the CHECK_TICK to a more sensible place
* Change the default turf to /turf/open/space/basic
* Skip loading default turfs
* No need for CreateSpace anymore
* Improve the comment
* SPACE OPTIMIZATION
* Don't need this instancing anymore either
* Forgot to drop this
* FASTER!!!!
* FASTER!!!!!!!!!
* Move this back, it makes no difference
* Fix a potential bug
* FAAAAAAASSSSSSTTTTTTTTERRRRRR!!!!
* open
* Ye be warned
* Better diagnostics + fixes
* Fewer false positives
* Bleh
* Remove the unecessary z-expansions
* Dynamic map loading prototype
* Get the title screen working
* Fixes the random pod bugs
* map configs
* Add back the asteroid
* Ignore the rotate file
* Fix some shit
* More shit fixes
* Babby's first json
* Cyberboss confirmed shitcoder
* Makes map rotation great again
* Moves the map defines
* Delete tgstation2.dm
* Convert runtimestation
* Convert PubbyStation
* Convert OmegaStation
* Convert Metastation
* Convert Delta Station
* Lol file extensions
* Reee line endings
* Take out the trash
* Fix loadallmaps.dm
* Deltaassssssstation?
* Fix travis
* Did I ASK you to change the line endings?
* Remove votable maps
* Cleanup
* Previous config. Load configs in New. Fix splash
* Fix this
* More generic minetype
* STOP TOUCHING THE LINE ENDINGS!!!
* Add some flexibility to /obj/screen/splash
* Fix the shit
* Update gitignore
* Smooth transition from last map
* Better check
* Missed this map rotate check
* Remove these checks too
* Prep for a future request by @kevinz000
* Fix z2 line endings
* Solution for custom .dms
* Rename some things, trim some fat
* Fixes default map_config
* Remove stale comment
* Delete the previous config after loading
* Don't try to open a non-existent file
* Delete the old rotator script
* And references to it
* Line endings
* More line endings
* Del checks from when m-config may have not existed
* LINE ENDINGS
* Update the default map config
* Map load error checking for what it's worth
* Small cleanup
* For case sensitive
* File systems
* Strip maprotate stuff from build script
* Spruce up the _maps tree. Compile some empty space
* Make travis simpler
* LINEARU ENDARU!!!
* tgstation TWO!!!
* Lowercase mine type
* Dummy dm's for testmerging
* Fixes job ordering
* If ghosts wanna watch the map load so be it
* Let them know what's going on
* Fixes z-transitions
* Optimize z expansion further
* Remove this old var
* Fix wiznerd teleportation
* Rmove butt
* Does the thing
* Moved temp configs to the data dir
* Un-touch gitignore
* Forgot templates.dm
* Forgot to include the includer
* Fair enough
* SILENCE IMBECILE!
* @Cyberboss still a fuckboi
* Speed things up a bit
* Fix a potential bug with the error message
* Whatever
* Reorganize the mapping code
* Finish this up
* Reee line endings
* FUCKING LINE ENDINGS
* LINARU ENDARU
* >PLS SET REPO LINE ENDINGS
* Comments
* Hoisted by my own travis.yml changes
