Replaces the asset subsystem's spritesheet generator with a rust-based
implementation (https://github.com/tgstation/rust-g/pull/160).
This is a rough port of
https://github.com/BeeStation/BeeStation-Hornet/pull/10404, but it
includes fixes for some cases I didn't catch that apply on TG.
(FWIW we've been using this system on prod for over a year and
encountered no major issues.)
`/datum/asset/spritesheet_batched`: A version of the spritesheet system
that collects a list of `/datum/universal_icon`s and sends them off to
rustg asynchronously, and the generation also runs on another thread, so
the game doesn't block during realize_spritesheet. The rust generation
is about 10x faster when it comes to actual icon generation, but the
biggest perk of the batched spritesheets is the caching system.
This PR notably does not convert a few things to the new spritesheet
generator.
- Species and antagonist icons in the preferences view because they use
getFlatIcon ~~which can't be converted to universal icons~~.
- Yes, this is still a *massive* cost to init, unfortunately. On Bee, I
actually enabled the 'legacy' cache on prod and development, which you
can see in my PR. That's why I added the 'clear cache' verb and the
`unregister()` procs, because it can force a regeneration at runtime. I
decided not to port this, since I think it would be detrimental to the
large amount of contributors here.
- It is *technically* possible to port parts of this to the uni_icon
system by making a uni_icon version of getFlatIcon. However, some
overlays use runtime-generated icons which are ~~completely unparseable
to IconForge, since they're stored in the RSC and don't exist as files
anywhere~~. This is most noticeable with things like hair (which blend
additively with the hair mask on the server, thus making them invisible
to `get_flat_uni_icon`). It also doesn't help that species and antag
icons will still need to generate a bunch of dummies and delete them to
even verify cache validity.
- It is actually possible to write the RSC icons to the filesystem
(using fcopy) and reference them in IconForge. However, I'm going to
wait on doing this until I port my GAGS implementation because it
requires GAGS to exist on the filesystem as well.
IconForge generates a cache based on the set of icons used, all
transform operations applied, and the source DMIs of each icon used
within the spritesheet. It can compare the hashes and invalidate the
cache automatically if any of these change. This means we can enable
caching on development, and have absolutely no downsides, because if
anything changes, the cache invalidates itself.
The caching has a mean cost of ~5ms and saves a lot of time compared to
generating the spritesheet, even with rust's faster generation. The main
downside is that the cache still requires building the list of icons and
their transforms, then json encoding it to send to rustg.
Here's an abbreviated example of a cache JSON. All of these need to
match for the cache to be valid. `input_hash` contains the transform
definitions for all the sprites in the spritesheet, so if the input to
iconforge changes, that hash catches it. The `sizes` and `sprites` are
loaded into DM.
```json
{
"input_hash": "99f1bc67d590e000",
"dmi_hashes": {
"icons/ui/achievements/achievements.dmi": "771200c75da11c62"
},
"sizes": [
"76x76"
],
"sprites": {
"achievement-rustascend": {
"size_id": "76x76",
"position": 1
}
},
"rustg_version": "3.6.0",
"dm_version": 1
}
```
Universal icons are just a collection of DMI, Icon State, and any icon
transformation procs you apply (blends, crops, scales). They can be
convered to DM icons via `to_icon()`. I've included an implementation of
GAGS that produces universal icons, allowing GAGS items to be converted
into them. IconForge can read universal icons and add them to
spritesheets. It's basically just a wrapper that reimplements BYOND icon
procs.
Converts some uses of md5asfile within legacy spritesheets to use
rustg_hash_file instead, improving the performance of their generation.
Fixes lizard body markings not showing in previews, and re-adds eyes to
the ethereal color preview. This is a side effect of IconForge having
*much* better error handling than DM icon procs. Invalid stuff that gets
passed around will error instead of silently doing nothing.
Changes the CSS used in legacy spritesheet generation to split
`background: url(...) no-repeat` into separate props. This is necessary
for WebView2, as IE treats these properties differently - adding
`background-color` to an icon object (as seen in the R&D console) won't
work if you don't split these out.
Deletes unused spritesheets and their associated icons (condiments
spritesheet, old PDA spritesheet)
If you press "Character Setup", the 10-13sec of lag is now approximately
0.5-2 seconds.
Tracy profile showing the time spent on get_asset_datum. I pressed the
preferences button during init on both branches. Do note that this was
ran with a smart cache HIT, so no generation occurred.
Much lower worst-case for /datum/asset/New (which includes
`create_spritesheets()` and `register()`)
Here's a look at the internal costs from rustg - as you can see
`generate_spritesheet()` is very fast:
**Before**
**After**
🆑
fix: Fixed lizard body markings and ethereal feature previews in the
preference menu missing some overlays.
refactor: Optimized spritesheet asset generation greatly using rustg
IconForge, greatly reducing post-initialization lag as well as reducing
init times and saving server computation.
config: Added 'smart' asset caching, for batched rustg IconForge
spritesheets. It is persistent and suitable for use on local, with
automatic invalidation.
add: Added admin verbs - Debug -> Clear Smart/Legacy Asset Cache for
spritesheets.
fix: Fixed R&D console icons breaking on WebView2/516
/🆑
## About The Pull Request
123 changed files and multiple crashes after writing broken regex, I
replaced most remains of direct spans with macros. This cleans up the
code and makes it easier to work with in general, see justification for
the original PR. I also fixed a bunch of broken and/or unclosed spans
here too.
I intentionally avoided replacing spans with multiple classes (in most
cases) and spans in the middle of strings as it would impact readability
(in my opinion at least) and could be done later if required.
## Why It's Good For The Game
Cleaner code, actually using our macros, fixes borked HTML in some
places. See original PR.
## Changelog
Nothing player-facing
# MAINTAINER - USE THE BUTTON THAT SAYS "MERGE MASTER" THEN SET THE PR
TO AUTO-MERGE! IT'S MUCH EASIER FOR ME TO FIX THINGS BEFORE THEY SKEW
RATHER THAN AFTER THE FACT.
## About The Pull Request
Hey there,
This took a while to do, but here's the gist:
Python file now regexes every file in `/code` except for those that have
some valid reason to be tacking on more global defines. Some of those
reasons are simply just that I don't have the time right now (doing what
you see in this PR took a few hours) to refactor and parse what should
belong and what should be thrown out. For the time being though, this PR
will at least _halt_ people making the mistake of not `#undef`ing any
files they `#define` "locally", or within the scope of a file.
Most people forget to do this and this leads to a lot of mess later on
due to how many variables can be unmanaged on the global level. I've
made this mistake, you've made this mistake, it's a common thing. Let's
automatically check for it so it can be fixed no-stress.
Scenarios this PR corrects:
* Forgetting to undef a define but undeffing others.
* Not undeffing any defines in your file.
* Earmarking a define as a "file local" define, but not defining it.
* Having a define be a "file local" define, but having it be used
elsewhere.
* Having a "local" define not even be in the file that it only shows up
in.
* Having a completely unused define*
(* I kept some of these because they seemed important... Others were
junked.)
## Why It's Good For The Game
If you wanna use it across multiple files, no reason to not make it a
global define (maybe there's a few reasons but let's assume that this is
the 95% case).
Let me know if you don't like how I re-arranged some of the defines and
how you'd rather see it be implemented, and I'd be happy to do that.
This was mostly just "eh does it need it or not" sorta stuff.
I used a pretty cool way to detect if we should use the standardized
GitHub "error" output, you can see the results of that here
https://github.com/san7890/bruhstation/actions/runs/4549766579/jobs/8022186846#step:7:792
## Changelog
Nothing that really concerns players.
(I fixed up all this stuff using vscode, no regexes beyond what you see
in the python script. sorry downstreams)
So i left over some basic `/whatever/proc/format` uses in the original
PR this fixes it.
Notable exceptions to the rule:
- Paths in add_verb/remove_verb, we need full path instead of a name
there to access verb metadata so we can't use proc ref macros there.
- regex.Replace, found out that it does not accept call by name. Instead
i added new REGEX_REPLACE_HANDLER so we can at least try to mark these.
There's still leftover global procs that do not use GLOBAL_PROC_REF but
they functionally equivalent so that's for later.
I don't see any reasonable way to grep for this. But if you got any
ideas please share.
Makes the code compatible with 515.1594+
Few simple changes and one very painful one.
Let's start with the easy:
* puts call behind `LIBCALL` define, so call_ext is properly used in 515
* Adds `NAMEOF_STATIC(_,X)` macro for nameof in static definitions since
src is now invalid there.
* Fixes tgui and devserver. From 515 onward the tmp3333{procid} cache
directory is not appened to base path in browser controls so we don't
check for it in base js and put the dev server dummy window file in
actual directory not the byond root.
* Renames the few things that had /final/ in typepath to ultimate since
final is a new keyword
And the very painful change:
`.proc/whatever` format is no longer valid, so we're replacing it with
new nameof() function. All this wrapped in three new macros.
`PROC_REF(X)`,`TYPE_PROC_REF(TYPE,X)`,`GLOBAL_PROC_REF(X)`. Global is
not actually necessary but if we get nameof that does not allow globals
it would be nice validation.
This is pretty unwieldy but there's no real alternative.
If you notice anything weird in the commits let me know because majority
was done with regex replace.
@tgstation/commit-access Since the .proc/stuff is pretty big change.
Co-authored-by: san7890 <the@san7890.com>
Co-authored-by: Mothblocks <35135081+Mothblocks@users.noreply.github.com>
* makes compile time icons not do expensive file io for asset gen
* better documentation on some asset vars and procs
* makes generate_and_hash_rsc_file() do fcopy() and passes down the chain
* implements msos suggestion to optimize get_icon_dmi_path()
* FINALLY IT IS DONE FUCK
* tip of the round into ooc
* adds cyopod chat messages to the info chat filter
* last one I promise probably, mob spawner messages to info filter
* holoparasite, round will start in x and keybinding conflict to respective filters
* removed motd from constants, made it infoplain surrounded instead
* changed info to infoplain
Replaces goonchat with a tgui based chat panel
Fixes#52898Fixes#52663
It is as fast as goonchat was (if not faster in certain circumstances), and is very extensible. It has all the necessary code for sorting messages into categories, which means that one of the next features will be multiple tab support.
Additional features that you will get with tgchat right now:
Massively faster server-side performance compared to goonchat, especially if batching multiple messages to one client.
Message persistence across rounds and reconnects. (All messages are stored client-side in IndexedDB)
More robust scroll tracking. If you scroll up, it will not change the scroll position on new messages like goonchat did.
Multiple message combining. (Currently set to combine up to 5 messages over last 5 seconds).
If using the highlighting feature, it highlights the whole message as well as the matching word.
"Now playing" widget, with preview of the song title, a knob for adjusting the volume and a stop button.
Architecture is as following:
```
to_chat() -+
|
SSchat
(queue, batching)
|
window.send_message()
|
v
+-------------+
| tgui-panel |
|+-----------+|
|| tgchat ||
|+-----------+|
+-------------+
```
Subsystem is basically goonchat, but without all the garbage that slows the servers down (string concatenation, double urlencoding, sanitizing, etc). Now, instead of all that, it's being slowed down by json_encode in /datum/tgui_window/proc/send_message, which IMO is completely worth it, and allows sending various templates and widgets to tgchat.
/datum/tgui_window abstracts the whole window away from you, establishes a nice message-passing interface between DM and JS, with two message queues on each side, automatically loads js/css assets for you, basically does everything. You as a developer only have to worry about sending/receiving messages and write javascript.
tgui-panel is a slimmed down version of tgui, and functions as a container for various widgets, and tgchat is one of them. It of course can be expanded with more stuff.
It's also a separate entry point and a JS bundle, so it's not bloating the main tgui bundle, and is currently sitting at about 230kB.
Rewrites the asset_cache system to handle sending assets to a CDN via a webroot.
see https://github.com/MrStonedOne/tgstation/blob/asset-cdn/code/modules/asset_cache/readme.md
Fixed a lot of bugs with assets, removed some dead code.
Changes:
Moved asset cache code to transport datums, the currently loaded one is located at SSassets.transport, asset cache calls made before the config is loaded use the simple browse_rsc transport.
Added subsystem call for when the config loads or reloads.
Added a webroot CDN asset transport. assets are saved to a file in a format based on the file's hash (currently md5).
Assets that don't use get_asset_url or get_url_mappings (such as browser assets referred to by static html files like changelog.html or static css files) can be saved to browse_rsc even when in cdn asset mode by setting legacy to TRUE on the datum returned by register_assets
Added a system for saving assets on a cdn in a hash based namespace (folder), assets within the same namespace will always be able to refer to each other by relative names. (used to allow cdn'ing font awesome without having to make something that regenerates it's css files.).
The simple/namespaced asset cache datum helper will handle generating a namespace composed of the combined md5 of everything in the same datum, as well as registering them properly.
Moved external resource from a snowflake loaded file to a config entry, added it to resources.txt
To ensure the system breaks in local testing in any situation that wouldn't work in cdn mode, the simple transport will mutate the filenames of non-legacy and non-namespaced assets and return this with get_asset_url.
Simple transport's passive send of all roundstart assets to all clients is now a config that defaults to off. this is to break race conditions during local testings from devs accidentally relying on this instead of using send() properly.
cl
refactor: Interface assets (js/css/images) can now be managed using an external webserver instead of byond's one at a time file transfer queue.
admin: Adds admin verb toggle-cdn that allows admins to disable the external webserver asset transport and revert to the old system. Useful if the webserver backing this goes down (thanks cloudflare).
config: New config file, resources.txt, (must be loaded by an $include statement from the main config)
server: The external_rsc_urls.txt config has been moved to the main config system.
/cl
Porting notes:
Interface webpages must refer to their assets (css/js/image/etc) by a generated url, or the asset must register itself as a legacy asset. The system is designed to break in localtest (on simple/legacy mode) in most situations that would break in cdn mode.
Requires latest tgui.
The webserver must set the proper CORS headers for font files or font awesome (and other fonts) won't load.
/tg/'s webserver config: https://gist.github.com/MrStonedOne/523388b2f161af832292d98a8aad0eae
