## About The Pull Request
This rewrites how our actions handle changelogs, and rewrites a bunch of
the changelog TGUI (in a separate file cause I don't feel like
massacring the upstream one). The gist of it is that our actions will
now create autochangelog files with bubber in the name, our changelog
compile will only compile those autochangelogs (not upstream ones if
they happen to be in there), and the compile will be done into separate
bubber changelog files in `html/changelogs/bubber_archive`. In game, the
UI will pull both archive files (upstream and ours) and combine them in
the changelog list, putting the codebase logo next to each change to
distinguish which one it's from.
~~NOTE: I'm opening this as a draft because I'm going to try and reset
our tg changelogs and then reapply all our historical changelogs into
the new bubber archive files, I'm just putting this up now so the code
can be reviewed and shit~~ done
## Why It's Good For The Game
For players this makes it easier to distinguish where a change came
from, and for developers this means we don't need to worry about
breaking the changelog when we do an upstream pull, we can completely
ignore it and just pull everything, and our tg archive files will be 1:1
with upstream with no shit from us added in.
## Proof Of Testing
<details>
<summary>Screenshots/Videos</summary>
</details>
## Changelog
🆑 TealSeer, LT3
refactor: refactored how changelogs are generated and displayed
/🆑
---------
Co-authored-by: lessthanthree <83487515+lessthnthree@users.noreply.github.com>
Replaces the asset subsystem's spritesheet generator with a rust-based
implementation (https://github.com/tgstation/rust-g/pull/160).
This is a rough port of
https://github.com/BeeStation/BeeStation-Hornet/pull/10404, but it
includes fixes for some cases I didn't catch that apply on TG.
(FWIW we've been using this system on prod for over a year and
encountered no major issues.)
`/datum/asset/spritesheet_batched`: A version of the spritesheet system
that collects a list of `/datum/universal_icon`s and sends them off to
rustg asynchronously, and the generation also runs on another thread, so
the game doesn't block during realize_spritesheet. The rust generation
is about 10x faster when it comes to actual icon generation, but the
biggest perk of the batched spritesheets is the caching system.
This PR notably does not convert a few things to the new spritesheet
generator.
- Species and antagonist icons in the preferences view because they use
getFlatIcon ~~which can't be converted to universal icons~~.
- Yes, this is still a *massive* cost to init, unfortunately. On Bee, I
actually enabled the 'legacy' cache on prod and development, which you
can see in my PR. That's why I added the 'clear cache' verb and the
`unregister()` procs, because it can force a regeneration at runtime. I
decided not to port this, since I think it would be detrimental to the
large amount of contributors here.
- It is *technically* possible to port parts of this to the uni_icon
system by making a uni_icon version of getFlatIcon. However, some
overlays use runtime-generated icons which are ~~completely unparseable
to IconForge, since they're stored in the RSC and don't exist as files
anywhere~~. This is most noticeable with things like hair (which blend
additively with the hair mask on the server, thus making them invisible
to `get_flat_uni_icon`). It also doesn't help that species and antag
icons will still need to generate a bunch of dummies and delete them to
even verify cache validity.
- It is actually possible to write the RSC icons to the filesystem
(using fcopy) and reference them in IconForge. However, I'm going to
wait on doing this until I port my GAGS implementation because it
requires GAGS to exist on the filesystem as well.
IconForge generates a cache based on the set of icons used, all
transform operations applied, and the source DMIs of each icon used
within the spritesheet. It can compare the hashes and invalidate the
cache automatically if any of these change. This means we can enable
caching on development, and have absolutely no downsides, because if
anything changes, the cache invalidates itself.
The caching has a mean cost of ~5ms and saves a lot of time compared to
generating the spritesheet, even with rust's faster generation. The main
downside is that the cache still requires building the list of icons and
their transforms, then json encoding it to send to rustg.
Here's an abbreviated example of a cache JSON. All of these need to
match for the cache to be valid. `input_hash` contains the transform
definitions for all the sprites in the spritesheet, so if the input to
iconforge changes, that hash catches it. The `sizes` and `sprites` are
loaded into DM.
```json
{
"input_hash": "99f1bc67d590e000",
"dmi_hashes": {
"icons/ui/achievements/achievements.dmi": "771200c75da11c62"
},
"sizes": [
"76x76"
],
"sprites": {
"achievement-rustascend": {
"size_id": "76x76",
"position": 1
}
},
"rustg_version": "3.6.0",
"dm_version": 1
}
```
Universal icons are just a collection of DMI, Icon State, and any icon
transformation procs you apply (blends, crops, scales). They can be
convered to DM icons via `to_icon()`. I've included an implementation of
GAGS that produces universal icons, allowing GAGS items to be converted
into them. IconForge can read universal icons and add them to
spritesheets. It's basically just a wrapper that reimplements BYOND icon
procs.
Converts some uses of md5asfile within legacy spritesheets to use
rustg_hash_file instead, improving the performance of their generation.
Fixes lizard body markings not showing in previews, and re-adds eyes to
the ethereal color preview. This is a side effect of IconForge having
*much* better error handling than DM icon procs. Invalid stuff that gets
passed around will error instead of silently doing nothing.
Changes the CSS used in legacy spritesheet generation to split
`background: url(...) no-repeat` into separate props. This is necessary
for WebView2, as IE treats these properties differently - adding
`background-color` to an icon object (as seen in the R&D console) won't
work if you don't split these out.
Deletes unused spritesheets and their associated icons (condiments
spritesheet, old PDA spritesheet)
If you press "Character Setup", the 10-13sec of lag is now approximately
0.5-2 seconds.
Tracy profile showing the time spent on get_asset_datum. I pressed the
preferences button during init on both branches. Do note that this was
ran with a smart cache HIT, so no generation occurred.
Much lower worst-case for /datum/asset/New (which includes
`create_spritesheets()` and `register()`)
Here's a look at the internal costs from rustg - as you can see
`generate_spritesheet()` is very fast:
**Before**
**After**
🆑
fix: Fixed lizard body markings and ethereal feature previews in the
preference menu missing some overlays.
refactor: Optimized spritesheet asset generation greatly using rustg
IconForge, greatly reducing post-initialization lag as well as reducing
init times and saving server computation.
config: Added 'smart' asset caching, for batched rustg IconForge
spritesheets. It is persistent and suitable for use on local, with
automatic invalidation.
add: Added admin verbs - Debug -> Clear Smart/Legacy Asset Cache for
spritesheets.
fix: Fixed R&D console icons breaking on WebView2/516
/🆑
## About The Pull Request
This PR reimplements https://github.com/tgstation/tgstation/pull/71538
atop `master`. Quoting the original PR:
> Every `icon_exists()` call will cache the entire file. Past me didn't
realise _why_ file opts were so expensive, but I do now. This is
immeasurably slower on a single call, and _significantly_ faster on
subsequent calls to the same file.
I attempted to handle some of the review comments that were posted
there, by splitting screaming functionality into its own proc.
* `if(icon_state in icon_states(file))` and `if(!(icon_state in
icon_states(file)))` were refactored to use `icon_exists(file,
icon_state)`.
* Where screaming was seemingly wanted (and where there wasn't a more
descriptive error inside the `if` block), I refactored them to use
`icon_exists_or_scream(file, icon_state)`
* The exception to the above was under
`/datum/unit_test/turf_icons/Run()` and
`/datum/unit_test/worn_icons/Run()`, where `icon_states()` was being
passed a mode flag. Given that this is only used in unit tests (where
performance isn't a priority), I opted to leave these be.
Additionally, I revised the documentation comment for
`/proc/icon_exists()`, as I felt it was a bit vague currently.
## Why It's Good For The Game
https://youtu.be/Z9G1Mf6TZRs
## Changelog
No player-facing changes (hopefully).
---------
Co-authored-by: SyncIt21 <110812394+SyncIt21@users.noreply.github.com>
## About The Pull Request
Adds a check for icon state existence in spritesheet insert, run during
unit tests.
## Why It's Good For The Game
Inserting a nonexistent icon state via `Insert()` will corrupt the
spritesheet it's inserted to, resulting in offsets being
incorrect—specifically, it results in the entire icon's contents being
inserted. This happened downstream on Doppler Shift and broke language
icons. I fixed the issue there but figured that there should be a check
upstream for it.
`if (!I || !length(icon_states(I))) // that direction or state doesn't
exist)` This check doesn't catch it, by the way. Since it returns the
entire icon file, `length(icon_states(I))` is >0. You could do
`length(icon_states) != 1`, but then it still wouldn't catch cases where
there's a single-icon-state icon *and* the icon_state is invalid. Boo. A
test like this is the best option.
I tried using the rust-g variant of icon_states and sort-of got it
working, but I figured it'd be too fragile to justify given that it
doesn't accept actual icon instances, only paths.
## About The Pull Request
Makes all spritesheets cache by default. This wasn't the case originally
because some spritesheets like vending machines relied on in world
state, but none of them do anymore because that's whack.
Also fixes a bug that would cause half completed caches to break other
stuff. This didn't happen in real gameplay, but would've happened if you
tried to change cachable on anything while you already had a tmp folder.
## Changelog
🆑
fix: Cut down a significant amount of time that caused the start of
rounds to lag.
/🆑
Investigating large lag spikes in round start and noticed a large amount
of overtime (3s worth) to queuedInsert.
It's vending machines and R&D, but let's keep an eye on this.
## About The Pull Request
Most of these are mine that I just forgot about, only one I think anyone
cares about is the one in mobs.dm about making delta time match
subsystem yielding, but I think it's a bad idea so it's gone
Oh also, replaces an old comment of mine with an actual explanation
(it's about the icon cache and shit)
## About The Pull Request
I was helping someone debug some weird bug with spritesheets a bit ago,
and I didn't like having to manually comment out all of the `fdel()`
stuff in order to help visualize what the potential issue might have
been with the spritesheets on either their DM-side generation or their
TGUI-level display. I decided to add a compile-time level flag that will
automatically copy over any generated spritesheet assets (css and pngs)
to the round-specific `data/logs` folder for analysis when a developer
should need it.
I also had to switch around some vars and make a few new ones to reduce
how copy-pasta it might get and ensure standardization/readability while
also being 0.001 times faster since we benefit from the string cache
(unprovable fact).
## Why It's Good For The Game
It's incredibly useful to see the actual flattened spritesheet itself
sometimes when you're doing this type of work and you keep getting odd
bugs here and there. Also saves headache from having to clear out the
temp `/data/spritesheets` folder every time you comment shit out, as
well as having an effective paper trail for A/B testing whatever
bullshit you've got going on.
## Changelog
Doesn't affect players.
The asset was being loaded, seeing that fully_generated is false, so it
attempts to rebuild. The rebuilding clears the existing file cache, and
fucks us.
Life is pain.
* Adds lazyloading to the asset subsystems
This currently applies only to spritesheets because of how monumentally
expensive they are.
If an asset is requested it will immediately be fully loaded, but
otherwise we slowly load them in with a separate subsystem.
This allows us to not hold up initialize with hair stuff. Saves roughly
33% (16 seconds with LOW_MEMORY_MODE) of initialize on my machine
My target is something closer to the 9 second init that had back in
2019, this is a good first step. Lets see how much more we can do yeah
lads?
Co-authored-by: san7890 <the@san7890.com>
This bug breaks disabling the cdn mid round if the cdn is broken, as well as enabling it mid round if it started the round disabled. some things fail back to byond transfers, but not all things.
* Move element to component, start UI, move assets into their own directory
* Complete UI
* Stop when another surgery is started
* Set your real zone since I forgot you actually need to start the surgery too
* Bring this back since I was just removing it as part of a cleanup for asset cache, but I can't prove it's not used anymore
* Remove unnecessary constructor I was using for something else
* Fix signal override
Preference asset creation, which while consistently created in early assets, can be requested at any time before then and often is, currently takes about 15 to 25 seconds to produce. Because of extremely hard to reproduce BYOND icon bugs, most of this is done on the same tick.
Lowering the cost of initialization itself is very tricky. Some of it we can theoretically optimize, such as creating humans for antagonists, others we can't, such as the raw cost of icon blending.
Furthermore, adding new icons later down the line would just increase this initialization time even more.
Instead of optimizing the asset creation, which is an uphill battle, this instead chooses to amortize the cost by caching preference assets created per git revision. This means that preference assets will be created, with their long delay, only once whenever the code changes.
This is done on a config, defaulting to on so that production needs no changes, as the whole point of these being made at runtime at all is that it keeps assets/art styles consistent, and PRs making subtle bugs that break preference generation in some way is not uncommon. On development, your git revision will stay the same until you commit, no matter what code changes you make.
Atomizes a much larger PR for another time...
There are typos in span and other html messages that causes them to not render correctly or at all.
Bug fixes
Converts those instances of span to use the macro
Bring _HELPERS/_lists.dm to latest standards by:
-Adding proper documentation and fixing existing one
-Giving vars proper names
-Procs now use snake case as per standard (many files that use those procs will be affected)
About The Pull Request
Rewrites the entire preferences menu in tgui. Rewrites the entire backend to be built upon datumized preferences, rather than constant additions to the preferences base datum.
Splits game preferences into its own window.
Antagonists are now split into their individual rulesets. You can now be a roundstart heretic without signing up for latejoin heretic, as an example.
This iteration matches parity, and provides very little new functionality, but adding anything new will be much easier.
Fixes#60823Fixes#28907Fixes#44887Fixes#59912Fixes#58458Fixes#59181
Major TODOs
Quirk icons, from @Fikou (with some slight adjustments from me)
Lore text, from @EOBGames (4/6, need moths and then ethereal lore from @AMonkeyThatCodes)
Heavy documentation on how one would add new preferences, species, jobs, etc
A lot of specialized testing so that people's real data don't get corrupted
Changelog
cl Mothblocks, Floyd on lots of the design
refactor: The preferences menu has been completely rewritten in tgui.
refactor: The "Stop Sounds" verb has been moved to OOC.
/cl
This pull request converts the changelog to TGUI.
Note: Old unused changelog files will be automatically removed on the next changelog run
Why It's Good For The Game
More consistent UI, ability to view all historic logs.
Changelog
cl Celotajs
refactor: Converted the changelog popup to TGUI
/cl
Replaces goonchat with a tgui based chat panel
Fixes#52898Fixes#52663
It is as fast as goonchat was (if not faster in certain circumstances), and is very extensible. It has all the necessary code for sorting messages into categories, which means that one of the next features will be multiple tab support.
Additional features that you will get with tgchat right now:
Massively faster server-side performance compared to goonchat, especially if batching multiple messages to one client.
Message persistence across rounds and reconnects. (All messages are stored client-side in IndexedDB)
More robust scroll tracking. If you scroll up, it will not change the scroll position on new messages like goonchat did.
Multiple message combining. (Currently set to combine up to 5 messages over last 5 seconds).
If using the highlighting feature, it highlights the whole message as well as the matching word.
"Now playing" widget, with preview of the song title, a knob for adjusting the volume and a stop button.
Architecture is as following:
```
to_chat() -+
|
SSchat
(queue, batching)
|
window.send_message()
|
v
+-------------+
| tgui-panel |
|+-----------+|
|| tgchat ||
|+-----------+|
+-------------+
```
Subsystem is basically goonchat, but without all the garbage that slows the servers down (string concatenation, double urlencoding, sanitizing, etc). Now, instead of all that, it's being slowed down by json_encode in /datum/tgui_window/proc/send_message, which IMO is completely worth it, and allows sending various templates and widgets to tgchat.
/datum/tgui_window abstracts the whole window away from you, establishes a nice message-passing interface between DM and JS, with two message queues on each side, automatically loads js/css assets for you, basically does everything. You as a developer only have to worry about sending/receiving messages and write javascript.
tgui-panel is a slimmed down version of tgui, and functions as a container for various widgets, and tgchat is one of them. It of course can be expanded with more stuff.
It's also a separate entry point and a JS bundle, so it's not bloating the main tgui bundle, and is currently sitting at about 230kB.
Rewrites the asset_cache system to handle sending assets to a CDN via a webroot.
see https://github.com/MrStonedOne/tgstation/blob/asset-cdn/code/modules/asset_cache/readme.md
Fixed a lot of bugs with assets, removed some dead code.
Changes:
Moved asset cache code to transport datums, the currently loaded one is located at SSassets.transport, asset cache calls made before the config is loaded use the simple browse_rsc transport.
Added subsystem call for when the config loads or reloads.
Added a webroot CDN asset transport. assets are saved to a file in a format based on the file's hash (currently md5).
Assets that don't use get_asset_url or get_url_mappings (such as browser assets referred to by static html files like changelog.html or static css files) can be saved to browse_rsc even when in cdn asset mode by setting legacy to TRUE on the datum returned by register_assets
Added a system for saving assets on a cdn in a hash based namespace (folder), assets within the same namespace will always be able to refer to each other by relative names. (used to allow cdn'ing font awesome without having to make something that regenerates it's css files.).
The simple/namespaced asset cache datum helper will handle generating a namespace composed of the combined md5 of everything in the same datum, as well as registering them properly.
Moved external resource from a snowflake loaded file to a config entry, added it to resources.txt
To ensure the system breaks in local testing in any situation that wouldn't work in cdn mode, the simple transport will mutate the filenames of non-legacy and non-namespaced assets and return this with get_asset_url.
Simple transport's passive send of all roundstart assets to all clients is now a config that defaults to off. this is to break race conditions during local testings from devs accidentally relying on this instead of using send() properly.
cl
refactor: Interface assets (js/css/images) can now be managed using an external webserver instead of byond's one at a time file transfer queue.
admin: Adds admin verb toggle-cdn that allows admins to disable the external webserver asset transport and revert to the old system. Useful if the webserver backing this goes down (thanks cloudflare).
config: New config file, resources.txt, (must be loaded by an $include statement from the main config)
server: The external_rsc_urls.txt config has been moved to the main config system.
/cl
Porting notes:
Interface webpages must refer to their assets (css/js/image/etc) by a generated url, or the asset must register itself as a legacy asset. The system is designed to break in localtest (on simple/legacy mode) in most situations that would break in cdn mode.
Requires latest tgui.
The webserver must set the proper CORS headers for font files or font awesome (and other fonts) won't load.
/tg/'s webserver config: https://gist.github.com/MrStonedOne/523388b2f161af832292d98a8aad0eae
Asset cache now caches the asset's md5 This should solve the high cpu usage issues with it. line by line profile suggests that the next hotspot is the asset log.
Simplied asset cache code, moved verify functionality to a flush_assets proc that blocks until a client has all currently sending assets. (sidenote: there is an argument for moving most of asset_cache's global procs onto the client, get_asset_datum and register_asset are really the only valid global procs.)
Re-added batched passive sends since it does speed up how quickly the asset cache pre-caches to clients.
The asset cache will now store json on the player's computer with info on what browser assets they have, all non-active assets (anything but .js(m) and .htm(l)) are reused in further connections. Browser assets in byond persist for an individual dream seeker instance, and are destoryed when the window is closed (so this only helps reconnects and round restarts) The data is stored in the client's asset folder to ensure its current and retrieved using javascript and sent back to the server using ajax(XHR).
The md5 of the asset files are generated on the server and stored on the client. It is used to validate the asset hasn't changed from a code update, and is not re-checked client side.
To ensure this can't be used by a malicious byond server to override javascript assets before redirecting people to /tg/ (where the attacker's javascript would then be allowed to run verbs and spoof topics) we do not mark javascript as pre-loaded when loading the client's asset cache json file on connection.
Other Changes
I moved some things around, the asset cache file was getting thicc:
Put new asset cache datums into code/modules/asset_cache/asset_list_items.dm
Find the asset cache datum abstract definitions inside code/modules/asset_cache/asset_list.dm
I fixed a bug where blocking asset sends would not block later calls to send the same asset while the send was still underway - todo: have it bind to the first send rather then initating its own.
The small file sent to the client to verify it got all pending asset sends will no longer use random names. This should keep the client's asset folder from exploding with hundreds of random htm files, much to the joy of oranges.
Passively loading assets no longer batches.
Unified the two procs.
Roxy
