## About The Pull Request
Replaces the asset subsystem's spritesheet generator with a rust-based
implementation (https://github.com/tgstation/rust-g/pull/160).
This is a rough port of
https://github.com/BeeStation/BeeStation-Hornet/pull/10404, but it
includes fixes for some cases I didn't catch that apply on TG.
(FWIW we've been using this system on prod for over a year and
encountered no major issues.)
### TG MAINTAINER NOTE
### Batched Spritesheets
`/datum/asset/spritesheet_batched`: A version of the spritesheet system
that collects a list of `/datum/universal_icon`s and sends them off to
rustg asynchronously, and the generation also runs on another thread, so
the game doesn't block during realize_spritesheet. The rust generation
is about 10x faster when it comes to actual icon generation, but the
biggest perk of the batched spritesheets is the caching system.
This PR notably does not convert a few things to the new spritesheet
generator.
- Species and antagonist icons in the preferences view because they use
getFlatIcon ~~which can't be converted to universal icons~~.
- Yes, this is still a *massive* cost to init, unfortunately. On Bee, I
actually enabled the 'legacy' cache on prod and development, which you
can see in my PR. That's why I added the 'clear cache' verb and the
`unregister()` procs, because it can force a regeneration at runtime. I
decided not to port this, since I think it would be detrimental to the
large amount of contributors here.
- It is *technically* possible to port parts of this to the uni_icon
system by making a uni_icon version of getFlatIcon. However, some
overlays use runtime-generated icons which are ~~completely unparseable
to IconForge, since they're stored in the RSC and don't exist as files
anywhere~~. This is most noticeable with things like hair (which blend
additively with the hair mask on the server, thus making them invisible
to `get_flat_uni_icon`). It also doesn't help that species and antag
icons will still need to generate a bunch of dummies and delete them to
even verify cache validity.
- It is actually possible to write the RSC icons to the filesystem
(using fcopy) and reference them in IconForge. However, I'm going to
wait on doing this until I port my GAGS implementation because it
requires GAGS to exist on the filesystem as well.
#### Caching
IconForge generates a cache based on the set of icons used, all
transform operations applied, and the source DMIs of each icon used
within the spritesheet. It can compare the hashes and invalidate the
cache automatically if any of these change. This means we can enable
caching on development, and have absolutely no downsides, because if
anything changes, the cache invalidates itself.
The caching has a mean cost of ~5ms and saves a lot of time compared to
generating the spritesheet, even with rust's faster generation. The main
downside is that the cache still requires building the list of icons and
their transforms, then json encoding it to send to rustg.
Here's an abbreviated example of a cache JSON. All of these need to
match for the cache to be valid. `input_hash` contains the transform
definitions for all the sprites in the spritesheet, so if the input to
iconforge changes, that hash catches it. The `sizes` and `sprites` are
loaded into DM.
```json
{
"input_hash": "99f1bc67d590e000",
"dmi_hashes": {
"icons/ui/achievements/achievements.dmi": "771200c75da11c62"
},
"sizes": [
"76x76"
],
"sprites": {
"achievement-rustascend": {
"size_id": "76x76",
"position": 1
}
},
"rustg_version": "3.6.0",
"dm_version": 1
}
```
### Universal Icons
Universal icons are just a collection of DMI, Icon State, and any icon
transformation procs you apply (blends, crops, scales). They can be
convered to DM icons via `to_icon()`. I've included an implementation of
GAGS that produces universal icons, allowing GAGS items to be converted
into them. IconForge can read universal icons and add them to
spritesheets. It's basically just a wrapper that reimplements BYOND icon
procs.
### Other Stuff
Converts some uses of md5asfile within legacy spritesheets to use
rustg_hash_file instead, improving the performance of their generation.
Fixes lizard body markings not showing in previews, and re-adds eyes to
the ethereal color preview. This is a side effect of IconForge having
*much* better error handling than DM icon procs. Invalid stuff that gets
passed around will error instead of silently doing nothing.
Changes the CSS used in legacy spritesheet generation to split
`background: url(...) no-repeat` into separate props. This is necessary
for WebView2, as IE treats these properties differently - adding
`background-color` to an icon object (as seen in the R&D console) won't
work if you don't split these out.
Deletes unused spritesheets and their associated icons (condiments
spritesheet, old PDA spritesheet)
## Why It's Good For The Game
If you press "Character Setup", the 10-13sec of lag is now approximately
0.5-2 seconds.
Tracy profile showing the time spent on get_asset_datum. I pressed the
preferences button during init on both branches. Do note that this was
ran with a smart cache HIT, so no generation occurred.
Much lower worst-case for /datum/asset/New (which includes
`create_spritesheets()` and `register()`)
Here's a look at the internal costs from rustg - as you can see
`generate_spritesheet()` is very fast:
### Comparison for a single spritesheet - chat spritesheet:
**Before**
**After**
## Changelog
🆑
fix: Fixed lizard body markings and ethereal feature previews in the
preference menu missing some overlays.
refactor: Optimized spritesheet asset generation greatly using rustg
IconForge, greatly reducing post-initialization lag as well as reducing
init times and saving server computation.
config: Added 'smart' asset caching, for batched rustg IconForge
spritesheets. It is persistent and suitable for use on local, with
automatic invalidation.
add: Added admin verbs - Debug -> Clear Smart/Legacy Asset Cache for
spritesheets.
fix: Fixed R&D console icons breaking on WebView2/516
/🆑
## About The Pull Request
Makes it so new dream types can be more easily added to the game with
effects within them beyond just some text being displayed if you want. I
added a dream that plays a random sound file available in the game files
to demonstrate how it's used. Mainly I wanted this out of the way for
another thing I'm working on.
I also made it so if the sound subsystem fails to reserve a channel it
throws a runtime since this happening is likely always going to result
in a bug, like it did while I was testing this.
## Changelog
🆑
add: New dream that plays sound at you
/🆑
## About The Pull Request
Converts all logging, excluding perf and investigate, to json.
I focused on making the system as easy to use and as easy to add new
categories as possible.
Due to issues related to logging to world at global creation logger is
now a byond real, which is created directly before Master
Log categories support versioning, secret flagging, and sub-category
filtering. Although all of this is entirely optional for coders.
If you ever want to add a new category and use it, all you need to do is
make the barebones category datum and the define.
I've kept existing procs such as log_game, and simply turned them into a
wrapper for Logger.Log(xxx, ...)
## Why It's Good For The Game
Makes processing and filtering logs much easier in the future, while
only minimally downgrading log crawling experience.
I am also working on a log viewer frontend for admin usage however that
will take a little bit longer to finish up.
Also makes special logging and data tracking much easier thanks to a
data list processing implementation and handling
## Changelog
🆑
server: All logs are now formatted in json, excluding perf and
investigations
/🆑
---------
Signed-off-by: GitHub <noreply@github.com>
Co-authored-by: tattle <66640614+dragomagol@users.noreply.github.com>
Co-authored-by: Kyle Spier-Swenson <kyleshome@gmail.com>
Co-authored-by: GoldenAlpharex <58045821+GoldenAlpharex@users.noreply.github.com>
I have been trying to load Tram station to show Lemon the smoother tram for a few hours now only to find out someone broke things big time.
I'm adding a sanitize_filepath proc to fix the flunky.
For those ignorant about the issue: Someone has recently modified the procs in the string_list file so they'd sanitize filenames, as a security precaution. Unfortunately, it turns out these procs are also used with filepaths as argument, which use the / as a delimiter between nodes.
Because the SANITIZE_FILENAME macro also purges delimiters (it's supposed to be used on filenames, not paths), this meant file paths were not being loaded correctly. Basically, it has broken map rotation, tcg and wound strings. At least the server hasn't been updated yet.
About The Pull Request
In my personal, subjective opinion; trialmins should not, in fact, be able to read and delete server/box configuration files on a whim.
cl
server: Patches multiple(?) arbitrary file related vulnerabilities
/cl
Bring _HELPERS/_lists.dm to latest standards by:
-Adding proper documentation and fixing existing one
-Giving vars proper names
-Procs now use snake case as per standard (many files that use those procs will be affected)
caught by monster at yogstation13/Yogstation#11703
probably not intended, 2^15 = 13 in dm so dont call this proc more than 13 times sequentially i guess
I have a suspicion that it doesn't matter assuming it gets caught by tick_usage being different but just in case
also fixes two other instances in code shown to me by mso
Done using this command sed -Ei 's/(\s*\S+)\s*\t+/\1 /g' code/**/*.dm
We have countless examples in the codebase with this style gone wrong, and defines and such being on hideously different levels of indentation. Fixing this to keep the alignment involves tainting the blames of code your PR doesn't need to be touching at all. And ultimately, it's hideous.
There are some files that this sed makes uglier. I can fix these when they are pointed out, but I believe this is ultimately for the greater good of readability. I'm more concerned with if any strings relied on this.
Hi codeowners!
Co-authored-by: Jared-Fogle <35135081+Jared-Fogle@users.noreply.github.com>
* Adds some file helpers for working with byond file references in external systems
file2filepath will convert icons and cache references into the filepath the file can be found at, or null if its a runtime created icon
generate_tmp_filepath will return a filepath to a file in the tmp file that you can save a runtime created cache reference (like an icon) to before sending it to external tools.
the asset cache's md5asfile will now use these systems to avoid saving files to the tmp that do not need to be saved to the tmp folder
* that would have been bad
icon2html will now always show the south direction unless passed a specific direction to show. Null can be explicitly passed with an atom to use the atom's direction. This is to work around icon() returning a blank icon if passed a direction that the icon doesn't define, instead of defaulting to south like the client does. This should also cut down on the number of assets generated by shift clicks
If icon2html is given an atom with an invalid icon state, it will use the atom's compile time icon_state, so smoothed icons can show their mapper's helper icon instead of the entire dmi.
Asset cache will now copy files to disk before md5ing them, to deal with byond not always returning the correct md5 for rsc references.
Replaces like 70-80% of 0 and such, as a side effect cleaned up a bunch of returns
Edit: Most left out ones are in mecha which should be done in mecha refactor already
Oh my look how clean it is
Co-authored-by: TiviPlus <TiviPlus>
Co-authored-by: Couls <coul422@gmail.com>
Rewrites the asset_cache system to handle sending assets to a CDN via a webroot.
see https://github.com/MrStonedOne/tgstation/blob/asset-cdn/code/modules/asset_cache/readme.md
Fixed a lot of bugs with assets, removed some dead code.
Changes:
Moved asset cache code to transport datums, the currently loaded one is located at SSassets.transport, asset cache calls made before the config is loaded use the simple browse_rsc transport.
Added subsystem call for when the config loads or reloads.
Added a webroot CDN asset transport. assets are saved to a file in a format based on the file's hash (currently md5).
Assets that don't use get_asset_url or get_url_mappings (such as browser assets referred to by static html files like changelog.html or static css files) can be saved to browse_rsc even when in cdn asset mode by setting legacy to TRUE on the datum returned by register_assets
Added a system for saving assets on a cdn in a hash based namespace (folder), assets within the same namespace will always be able to refer to each other by relative names. (used to allow cdn'ing font awesome without having to make something that regenerates it's css files.).
The simple/namespaced asset cache datum helper will handle generating a namespace composed of the combined md5 of everything in the same datum, as well as registering them properly.
Moved external resource from a snowflake loaded file to a config entry, added it to resources.txt
To ensure the system breaks in local testing in any situation that wouldn't work in cdn mode, the simple transport will mutate the filenames of non-legacy and non-namespaced assets and return this with get_asset_url.
Simple transport's passive send of all roundstart assets to all clients is now a config that defaults to off. this is to break race conditions during local testings from devs accidentally relying on this instead of using send() properly.
cl
refactor: Interface assets (js/css/images) can now be managed using an external webserver instead of byond's one at a time file transfer queue.
admin: Adds admin verb toggle-cdn that allows admins to disable the external webserver asset transport and revert to the old system. Useful if the webserver backing this goes down (thanks cloudflare).
config: New config file, resources.txt, (must be loaded by an $include statement from the main config)
server: The external_rsc_urls.txt config has been moved to the main config system.
/cl
Porting notes:
Interface webpages must refer to their assets (css/js/image/etc) by a generated url, or the asset must register itself as a legacy asset. The system is designed to break in localtest (on simple/legacy mode) in most situations that would break in cdn mode.
Requires latest tgui.
The webserver must set the proper CORS headers for font files or font awesome (and other fonts) won't load.
/tg/'s webserver config: https://gist.github.com/MrStonedOne/523388b2f161af832292d98a8aad0eae
* Unicode support Part 2 -- copytext()
This is the transition of all copytext() calls to be unicode aware and also some nearby calls in the same functions. Most things are just replacing copytext() with copytext_char() as a terrible character limiter but a few others were slightly more involved.
I replaced a ton of
````
var/something = sanitize(input())
something = copytext(something, 1, MAX_MESSAGE_LEN)
````
with a single stripped_input() call. stripped_input() already calls html_encode(), trim(), and some other sanitization so there shouldn't be any major issues there.
This is still VERY rough btw; DNA is a mess, the status displays are complete ass, there's a copytext() in code\datums\shuttles.dm that I'm not sure what to do with, and I didn't touch anything in the tools folder. I haven't tested this much at all yet, I only got it to compile earlier this morning. There's also likely to be weird bugs until I get around to fixing length(), findtext(), and the rest of the string procs.
* Makes the code functional
* Assume color hex strings are always # followed by ascii.
Properly encodes and decodes the stuff in mob_helpers.dm which fixes some issues there.
* Removes ninjaspeak since it's unused
* Adds sorting to most input() lists.
* Sorted some global lists, added more input sorting
* Should now use correct sort everywhere.
* compiles
* Last fixes.
* Adds new helper, DisplayTimeText
* Removed unused define
Thought it would've been nice to have for the future, but I guess it's
fine to go without it.
* CBB
* Revamp
* Early returns
* More cleanup
* Proc cleanup
* Makes fraction only show if seconds is < 1
* Last cleanup
* Revert
* Corrected incorrect round time.
Dunno how nobody ever caught that it was incorrect on live servers, gg.
* Add the system for managed global variables
* Travis ban old globals
* So you CAN inline proccall, that's neat
* Fix that
* master.dm
* Remove the hack procs
* Move InitGlobals to the proper spot
* configuration.dm
* Fix the missing pre-slash
* clockcult.dm
* This is probably for the best
* Doy
* Fix shit
* Rest of the DEFINES tree
* Fix
* Use global. for access
* Update find_references_in_globals
Always hated that proc
Whoever made it must've bee a r e a l idiot...
* __HELPERS tree
* Move global initialization to master.
Fix the declaration
* database.dm
* Dat newline
* I said DECLARATIVE order!
* Here's something you can chew on @Iamgoofball
* game_modes.dm
* Fix this
* genetics.dm
* flavor_misc.dm
* More stuff
* Do it mso's way. Keep the controllers as global
* Make master actually see it
* Fix
* Finish _globalvars/lists
* Finish the rest of the _globalvars tree
* This is weird
* Migrate the controllers
* SLOTH -> GLOB
* Lighting globals
* round_start_time -> ticker
* PAI card list -> pai SS
* record_id_num -> static
* Diseases list -> SSdisease
* More disease globals to the SS
* More disease stuff
* Emote list
* Better and better
* Bluh
* So much stuff
* Ahh
* Wires
* dview
* station_areas
* Teleportlocs
* blood_splatter_icons
* Stuff and such
* More stuff
* RAD IO
* More stuff and such
* Blob shit
* Changeling stuff
* Add "Balance" to changelogs
* Balance for changelog compiler + Auto Tagging
* Update the PR template
* hivemind_bank
* Bip
* sacrificed
* Good shit
* Better define
* More cult shit
* Devil shit
* Gang shit
* > borers
Fix shit
* Rename the define
* Nuke
* Objectives
* Sandbox
* Multiverse sword
* Announce systems
* Stuff and such
* TC con
* Airlock
* doppllllerrrrrr
* holopads
* Shut up byond you inconsistent fuck
* Sneaky fuck
* Burp
* Bip
* Fixnshit
* Port without regard
* askdlfjs;
* asdfjasoidojfi
* Protected globals and more
* SO MANY
* ajsimkvahsaoisd
* akfdsiaopwimfeoiwafaw
* gsdfigjosidjfgiosdg
* AHHHHHHHHHHHHHHHHHHHHHHH!!!!!
* facerolll
* ASDFASDFASDF
* Removes the unused parts of dmm_suite
* WIP
* Fix quote
* asdfjauwfnkjs
* afwlunhskjfda
* asfjlaiwuefhaf
* SO CLOSE
* wwwweeeeeewwwww
* agdgmoewranwg
* HOLY MOTHER OF FUCK AND THATS JUST HALF THE JOB?!?
* Fix syntax errors
* 100 errors
* Another 100
* So many...
* Ugh
* More shit
* kilme
* Stuuuuuufffff
* ajrgmrlshio;djfa;sdkl
* jkbhkhjbmjvjmh
* soi soi soi
* butt
* TODAY WE LEARNED THAT GLOBAL AND STATIC ARE THE EXACT SAME FUCKING THING
* lllllllllllllllllllllllllllllllllllllllllll
* afsdijfiawhnflnjhnwsdfs
* yugykihlugk,kj
* time to go
* STUFFF!!!
* AAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHH!!!!!!!!!!!!!!!!!!!!!!!
* ngoaijdjlfkamsdlkf
* Break time
* aufjsdklfalsjfi
* CONTROL KAY AND PRAY
* IT COMPILEELEELELAKLJFKLDAFJLKFDJLADKJHFLJKAJGAHIEJALDFJ!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
* Goteem
* Fix testing mode
* This does not belong in this PR
* Convert it to a controller
* Eh, fuck this option
* Revert controllerization Ill do it some other time
* Fix
* Working controllerization
* FOR THE LOVE OF CHRIST PROTECT THE LOGS
* Protect admins and deadmins
* Use the inbuilt proc
* Starting out on our tgui journey
* God tgui, why do you need to be updated
You're dynamically generated anyway
* Commit of stuff
* Further progress is being made
* Everyone loves buildscripts
* Further modifications to my incredible running script
* Starting to modify the minimap code to be better
* It's going well thusfar, I guess
* What have I done
* RIP minimap
* FUN FUN FUN FUN FUN
* Adds shuttle_manipulator.dmi
MUH HOLOGRAMS
* Is it done? IS IT OVER
* Peer review
* Some bug fixes
* Makes that damn greentext shut up
* Shuttle registration
* Made the Emergency Escape Bar more robust
No climbing on the bar.
* Do not stare into the operation end of the device
* Compile shame
* THE MOST DUMB
* Passive barmaiden
* Emergency shuttle memes
* MORE SAFETY CODE
* Fancy shuttle manipulator icons
* Smoothing it out
* We are going to have a lot of fun with this one
* Independent blobbernauts
* WABBAJACK WABBAJACK
* Message for attempting to authenticate old style
* Angry alert noise is back
* Revert "Independent blobbernauts"
This reverts commit 34d6af7c9c88cfc2864990cb37b586bb90163dd3.
* No parrot sleep plz
* Moves the special shuttle stuff to special.dm
* No Bartender like a Centcom Bartender
* Non-controversial map changes
- Backup shuttle moved closer to Centcom main structure, docking tube
created
- Moved shuttle import landmark to above Centcom main building
- Added shuttle displays to Conference Room
- Squashed the Chapel a bit in Metastation
- Made the docking port on Z2 massive
- Made the docking port on Metastation a lot larger
* Hacks and slashes at Box
A bunch of things are extended and squashed so Box shuttle dock can
support the MetaStation emergency shuttle.
* Some Metastationshit
* Never ending changes
* Wabbajack to TGM
* Modified the bar, I think that's all of them
* Stops Barmaiden wandering around
* More code review
* Whitspace, the bane of us all
* DIE WHITESPACE DIE
This requires a 508 beta version to use. If Travis fails this he's a bitch
Exceptions will generate a stack trace, which is way easier to see and more helpful in actually solving this kind of crap. Also logs all the arguments, src, line, and file automatically.
Removed any dubiously helpful information in the exception names so the runtime condenser won't see each one as a different runtime. If the information is critical to solve these bugs (camera one maybe?), then I'll just make these warnings.
Thrown exceptions crash the currently running proc. Yes that means there's useless returns in a bunch of these, sue me. spawn()'s are to let the proc continue.
Almost all of these are difficult to trigger, but I did test playsound. And frankly even if they do cause bugs by crashing procs, big whoop
fixed giveruntimelog getruntimelog and getserverlog
Any admin with ADMIN rights can give anybody permission to view runtimes simply by typing .giveruntimelog The person given permission can then type .getruntimelog
Admins can type .getruntimelog without giving themselves permissions first
Moved a lot of the copypasta code into helper procs in __HELPERS/files.dm. There is one which allows a client to browse through folders on the server (in this case to look for logfiles). You can now also return to the directory you started at.
Note, for some weird reason, runtimes will no longer show in dreamdeamon. If this is a massive problem I can make it a compile option or something (or you can comment out the line if you're a coder). I know this is a massive pain but it sort of removes much of the effort in getting runtimes from other servers, since they all now have their runtimes saved in a way coders can easily access. It also sort of means we don't have to rely on people remembering to log runtimes and such. And all logs will be organised by month.
git-svn-id: http://tgstation13.googlecode.com/svn/trunk@5466 316c924e-a436-60f5-8080-3fe189b3f50e
Integrated the sql and standard admin_rank stuff a little. Still needs some work.
Permissions Panel has a sexy floating search bar. The scripts and stylesheets are external files which will be sent to every client at connect, so they can be used in any panel you wish with no additional overheads.
If there are any bugs with the permissions panel (particularly the search bar) please let me know. thanks.
rights2text now has a seperator argument. It defaults to "", but can be any string, for instance "<br>"
git-svn-id: http://tgstation13.googlecode.com/svn/trunk@5080 316c924e-a436-60f5-8080-3fe189b3f50e
>Replaced dd_text2list, dd_text2listcase, tg_text2listcase and tg_text2list with text2list and text2listEx. text2list will return a list of each and every character in the string if you set separator=""
>added return_file_text(filepath) which returns text from a file after doing some checks: does the file exist? is the file empty? It prints helpful error messages to the world.log if it runs into problems
>Replaced dd_file2list(filepath, seperator) with file2list(filepath, seperator). It just calls text2list(return_file_text(filepath), seperator). rather than copypasta
>Replaced time_stamp() so it's not as retarded
>Lots of the world setup stuff uses file2list now, rather than file2text -> sanity -> text2list
>Added error() warning() testing() procs. These print messages to world.log with a prefix. e.g. ## ERROR: msg.
git-svn-id: http://tgstation13.googlecode.com/svn/trunk@4948 316c924e-a436-60f5-8080-3fe189b3f50e
