Files
Bubberstation/code/modules/asset_cache/asset_list.dm
itsmeow cc335e7e9e IconForge: rust-g Spritesheet Generation (#89478)
## About The Pull Request

Replaces the asset subsystem's spritesheet generator with a rust-based
implementation (https://github.com/tgstation/rust-g/pull/160).

This is a rough port of
https://github.com/BeeStation/BeeStation-Hornet/pull/10404, but it
includes fixes for some cases I didn't catch that apply on TG.

(FWIW we've been using this system on prod for over a year and
encountered no major issues.)

### TG MAINTAINER NOTE


![image](https://github.com/user-attachments/assets/53bd2b44-9bb5-42d2-b33f-093651edebc0)

### Batched Spritesheets

`/datum/asset/spritesheet_batched`: A version of the spritesheet system
that collects a list of `/datum/universal_icon`s and sends them off to
rustg asynchronously, and the generation also runs on another thread, so
the game doesn't block during realize_spritesheet. The rust generation
is about 10x faster when it comes to actual icon generation, but the
biggest perk of the batched spritesheets is the caching system.

This PR notably does not convert a few things to the new spritesheet
generator.

- Species and antagonist icons in the preferences view because they use
getFlatIcon ~~which can't be converted to universal icons~~.
- Yes, this is still a *massive* cost to init, unfortunately. On Bee, I
actually enabled the 'legacy' cache on prod and development, which you
can see in my PR. That's why I added the 'clear cache' verb and the
`unregister()` procs, because it can force a regeneration at runtime. I
decided not to port this, since I think it would be detrimental to the
large amount of contributors here.
- It is *technically* possible to port parts of this to the uni_icon
system by making a uni_icon version of getFlatIcon. However, some
overlays use runtime-generated icons which are ~~completely unparseable
to IconForge, since they're stored in the RSC and don't exist as files
anywhere~~. This is most noticeable with things like hair (which blend
additively with the hair mask on the server, thus making them invisible
to `get_flat_uni_icon`). It also doesn't help that species and antag
icons will still need to generate a bunch of dummies and delete them to
even verify cache validity.
- It is actually possible to write the RSC icons to the filesystem
(using fcopy) and reference them in IconForge. However, I'm going to
wait on doing this until I port my GAGS implementation because it
requires GAGS to exist on the filesystem as well.

#### Caching

IconForge generates a cache based on the set of icons used, all
transform operations applied, and the source DMIs of each icon used
within the spritesheet. It can compare the hashes and invalidate the
cache automatically if any of these change. This means we can enable
caching on development, and have absolutely no downsides, because if
anything changes, the cache invalidates itself.

The caching has a mean cost of ~5ms and saves a lot of time compared to
generating the spritesheet, even with rust's faster generation. The main
downside is that the cache still requires building the list of icons and
their transforms, then json encoding it to send to rustg.

Here's an abbreviated example of a cache JSON. All of these need to
match for the cache to be valid. `input_hash` contains the transform
definitions for all the sprites in the spritesheet, so if the input to
iconforge changes, that hash catches it. The `sizes` and `sprites` are
loaded into DM.

```json
{
	"input_hash": "99f1bc67d590e000",
	"dmi_hashes": {
		"icons/ui/achievements/achievements.dmi": "771200c75da11c62"
	},
	"sizes": [
		"76x76"
	],
	"sprites": {
		"achievement-rustascend": {
			"size_id": "76x76",
			"position": 1
		}
	},
	"rustg_version": "3.6.0",
	"dm_version": 1
}
```

### Universal Icons

Universal icons are just a collection of DMI, Icon State, and any icon
transformation procs you apply (blends, crops, scales). They can be
convered to DM icons via `to_icon()`. I've included an implementation of
GAGS that produces universal icons, allowing GAGS items to be converted
into them. IconForge can read universal icons and add them to
spritesheets. It's basically just a wrapper that reimplements BYOND icon
procs.

### Other Stuff

Converts some uses of md5asfile within legacy spritesheets to use
rustg_hash_file instead, improving the performance of their generation.

Fixes lizard body markings not showing in previews, and re-adds eyes to
the ethereal color preview. This is a side effect of IconForge having
*much* better error handling than DM icon procs. Invalid stuff that gets
passed around will error instead of silently doing nothing.

Changes the CSS used in legacy spritesheet generation to split
`background: url(...) no-repeat` into separate props. This is necessary
for WebView2, as IE treats these properties differently - adding
`background-color` to an icon object (as seen in the R&D console) won't
work if you don't split these out.

Deletes unused spritesheets and their associated icons (condiments
spritesheet, old PDA spritesheet)

## Why It's Good For The Game

If you press "Character Setup", the 10-13sec of lag is now approximately
0.5-2 seconds.

Tracy profile showing the time spent on get_asset_datum. I pressed the
preferences button during init on both branches. Do note that this was
ran with a smart cache HIT, so no generation occurred.


![image](https://github.com/user-attachments/assets/3efa71ab-972b-4f5a-acab-0892496ef999)

Much lower worst-case for /datum/asset/New (which includes
`create_spritesheets()` and `register()`)


![image](https://github.com/user-attachments/assets/9ad8ceee-7bd6-4c48-b5f3-006520f527ef)

Here's a look at the internal costs from rustg - as you can see
`generate_spritesheet()` is very fast:


![image](https://github.com/user-attachments/assets/e6892c28-8c31-4af5-96d4-501e966d0ce9)

### Comparison for a single spritesheet - chat spritesheet:

**Before**


![image](https://github.com/user-attachments/assets/cbd65787-42ba-4278-a45c-bd3d538da986)

**After**


![image](https://github.com/user-attachments/assets/d750899a-bd07-4b57-80fb-420fcc0ae416)

## Changelog

🆑
fix: Fixed lizard body markings and ethereal feature previews in the
preference menu missing some overlays.
refactor: Optimized spritesheet asset generation greatly using rustg
IconForge, greatly reducing post-initialization lag as well as reducing
init times and saving server computation.
config: Added 'smart' asset caching, for batched rustg IconForge
spritesheets. It is persistent and suitable for use on local, with
automatic invalidation.
add: Added admin verbs - Debug -> Clear Smart/Legacy Asset Cache for
spritesheets.
fix: Fixed R&D console icons breaking on WebView2/516
/🆑
2025-03-03 14:58:27 +01:00

276 lines
9.8 KiB
Plaintext

//These datums are used to populate the asset cache, the proc "register()" does this.
//Place any asset datums you create in asset_list_items.dm
//all of our asset datums, used for referring to these later
GLOBAL_LIST_EMPTY(asset_datums)
//get an assetdatum or make a new one
//does NOT ensure it's filled, if you want that use get_asset_datum()
/proc/load_asset_datum(type)
return GLOB.asset_datums[type] || new type()
/proc/get_asset_datum(type)
var/datum/asset/loaded_asset = GLOB.asset_datums[type] || new type()
return loaded_asset.ensure_ready()
/datum/asset
var/_abstract = /datum/asset
var/cached_serialized_url_mappings
var/cached_serialized_url_mappings_transport_type
/// Whether or not this asset should be loaded in the "early assets" SS
var/early = FALSE
/// Whether or not this asset can be cached across rounds of the same commit under the `CACHE_ASSETS` config.
/// This is not a *guarantee* the asset will be cached. Not all asset subtypes respect this field, and the
/// config can, of course, be disabled.
/// Disable this if your asset can change between rounds on the same exact version of the code.
var/cross_round_cachable = FALSE
/datum/asset/New()
GLOB.asset_datums[type] = src
register()
/// Stub that allows us to react to something trying to get us
/// Not useful here, more handy for sprite sheets
/datum/asset/proc/ensure_ready()
return src
/// Stub to hook into if your asset is having its generation queued by SSasset_loading
/datum/asset/proc/queued_generation()
CRASH("[type] inserted into SSasset_loading despite not implementing /proc/queued_generation")
/datum/asset/proc/get_url_mappings()
return list()
/// Returns a cached tgui message of URL mappings
/datum/asset/proc/get_serialized_url_mappings()
if (isnull(cached_serialized_url_mappings) || cached_serialized_url_mappings_transport_type != SSassets.transport.type)
cached_serialized_url_mappings = TGUI_CREATE_MESSAGE("asset/mappings", get_url_mappings())
cached_serialized_url_mappings_transport_type = SSassets.transport.type
return cached_serialized_url_mappings
/datum/asset/proc/register()
return
/datum/asset/proc/send(client)
return
/// Returns whether or not the asset should attempt to read from cache
/datum/asset/proc/should_refresh()
return !cross_round_cachable || !CONFIG_GET(flag/cache_assets)
/// Immediately regenerate the asset, overwriting any cache.
/datum/asset/proc/regenerate()
unregister()
cached_serialized_url_mappings = null
cached_serialized_url_mappings_transport_type = null
register()
/// Unregisters any assets from the transport.
/datum/asset/proc/unregister()
CRASH("unregister() not implemented for asset [type]!")
/// Simply takes any generated file and saves it to the round-specific /logs folder. Useful for debugging potential issues with spritesheet generation/display.
/// Only called when the SAVE_SPRITESHEETS config option is uncommented.
/datum/asset/proc/save_to_logs(file_name, file_location)
var/asset_path = "[GLOB.log_directory]/generated_assets/[file_name]"
fdel(asset_path) // just in case, sadly we can't use rust_g stuff here.
fcopy(file_location, asset_path)
/// If you don't need anything complicated.
/datum/asset/simple
_abstract = /datum/asset/simple
/// list of assets for this datum in the form of:
/// asset_filename = asset_file. At runtime the asset_file will be
/// converted into a asset_cache datum.
var/assets = list()
/// Set to true to have this asset also be sent via the legacy browse_rsc
/// system when cdn transports are enabled?
var/legacy = FALSE
/// TRUE for keeping local asset names when browse_rsc backend is used
var/keep_local_name = FALSE
/datum/asset/simple/register()
for(var/asset_name in assets)
var/datum/asset_cache_item/ACI = SSassets.transport.register_asset(asset_name, assets[asset_name])
if (!ACI)
log_asset("ERROR: Invalid asset: [type]:[asset_name]:[ACI]")
continue
if (legacy)
ACI.legacy = legacy
if (keep_local_name)
ACI.keep_local_name = keep_local_name
assets[asset_name] = ACI
/datum/asset/simple/send(client)
. = SSassets.transport.send_assets(client, assets)
/datum/asset/simple/get_url_mappings()
. = list()
for (var/asset_name in assets)
.[asset_name] = SSassets.transport.get_asset_url(asset_name, assets[asset_name])
/datum/asset/simple/unregister()
for (var/asset_name in assets)
SSassets.transport.unregister_asset(asset_name)
// For registering or sending multiple others at once
/datum/asset/group
_abstract = /datum/asset/group
var/list/children
/datum/asset/group/register()
for(var/type in children)
load_asset_datum(type)
/datum/asset/group/send(client/C)
for(var/type in children)
var/datum/asset/A = get_asset_datum(type)
. = A.send(C) || .
/datum/asset/group/get_url_mappings()
. = list()
for(var/type in children)
var/datum/asset/A = get_asset_datum(type)
. += A.get_url_mappings()
/datum/asset/group/unregister()
for (var/type in children)
var/datum/asset/A = get_asset_datum(type)
A.unregister()
/datum/asset/changelog_item
_abstract = /datum/asset/changelog_item
var/item_filename
/datum/asset/changelog_item/New(date)
item_filename = SANITIZE_FILENAME("[date].yml")
SSassets.transport.register_asset(item_filename, file("html/changelogs/archive/" + item_filename))
/datum/asset/changelog_item/send(client)
if (!item_filename)
return
. = SSassets.transport.send_assets(client, item_filename)
/datum/asset/changelog_item/get_url_mappings()
if (!item_filename)
return
. = list("[item_filename]" = SSassets.transport.get_asset_url(item_filename))
/datum/asset/changelog_item/unregister()
if (!item_filename)
return
SSassets.transport.unregister_asset(item_filename)
//Generates assets based on iconstates of a single icon
/datum/asset/simple/icon_states
_abstract = /datum/asset/simple/icon_states
var/icon
var/list/directions = list(SOUTH)
var/frame = 1
var/movement_states = FALSE
var/prefix = "default" //asset_name = "[prefix].[icon_state_name].png"
var/generic_icon_names = FALSE //generate icon filenames using generate_asset_name() instead the above format
/datum/asset/simple/icon_states/register(_icon = icon)
for(var/icon_state_name in icon_states(_icon))
for(var/direction in directions)
var/asset = icon(_icon, icon_state_name, direction, frame, movement_states)
if (!asset)
continue
asset = fcopy_rsc(asset) //dedupe
var/prefix2 = (directions.len > 1) ? "[dir2text(direction)]." : ""
var/asset_name = SANITIZE_FILENAME("[prefix].[prefix2][icon_state_name].png")
if (generic_icon_names)
asset_name = "[generate_asset_name(asset)].png"
SSassets.transport.register_asset(asset_name, asset)
/datum/asset/simple/icon_states/multiple_icons
_abstract = /datum/asset/simple/icon_states/multiple_icons
var/list/icons
/datum/asset/simple/icon_states/multiple_icons/register()
for(var/i in icons)
..(i)
/// Namespace'ed assets (for static css and html files)
/// When sent over a cdn transport, all assets in the same asset datum will exist in the same folder, as their plain names.
/// Used to ensure css files can reference files by url() without having to generate the css at runtime, both the css file and the files it depends on must exist in the same namespace asset datum. (Also works for html)
/// For example `blah.css` with asset `blah.png` will get loaded as `namespaces/a3d..14f/f12..d3c.css` and `namespaces/a3d..14f/blah.png`. allowing the css file to load `blah.png` by a relative url rather then compute the generated url with get_url_mappings().
/// The namespace folder's name will change if any of the assets change. (excluding parent assets)
/datum/asset/simple/namespaced
_abstract = /datum/asset/simple/namespaced
/// parents - list of the parent asset or assets (in name = file assoicated format) for this namespace.
/// parent assets must be referenced by their generated url, but if an update changes a parent asset, it won't change the namespace's identity.
var/list/parents = list()
/datum/asset/simple/namespaced/register()
if (legacy)
assets |= parents
var/list/hashlist = list()
var/list/sorted_assets = sort_list(assets)
for (var/asset_name in sorted_assets)
var/datum/asset_cache_item/ACI = new(asset_name, sorted_assets[asset_name])
if (!ACI?.hash)
log_asset("ERROR: Invalid asset: [type]:[asset_name]:[ACI]")
continue
hashlist += ACI.hash
sorted_assets[asset_name] = ACI
var/namespace = md5(hashlist.Join())
for (var/asset_name in parents)
var/datum/asset_cache_item/ACI = new(asset_name, parents[asset_name])
if (!ACI?.hash)
log_asset("ERROR: Invalid asset: [type]:[asset_name]:[ACI]")
continue
ACI.namespace_parent = TRUE
sorted_assets[asset_name] = ACI
for (var/asset_name in sorted_assets)
var/datum/asset_cache_item/ACI = sorted_assets[asset_name]
if (!ACI?.hash)
log_asset("ERROR: Invalid asset: [type]:[asset_name]:[ACI]")
continue
ACI.namespace = namespace
assets = sorted_assets
..()
/// Get a html string that will load a html asset.
/// Needed because byond doesn't allow you to browse() to a url.
/datum/asset/simple/namespaced/proc/get_htmlloader(filename)
return url2htmlloader(SSassets.transport.get_asset_url(filename, assets[filename]))
/// A subtype to generate a JSON file from a list
/datum/asset/json
_abstract = /datum/asset/json
/// The filename, will be suffixed with ".json"
var/name
/datum/asset/json/send(client)
return SSassets.transport.send_assets(client, "[name].json")
/datum/asset/json/get_url_mappings()
return list(
"[name].json" = SSassets.transport.get_asset_url("[name].json"),
)
/datum/asset/json/register()
var/filename = "data/[name].json"
fdel(filename)
rustg_file_write(json_encode(generate()), filename)
SSassets.transport.register_asset("[name].json", fcopy_rsc(filename))
fdel(filename)
/// Returns the data that will be JSON encoded
/datum/asset/json/proc/generate()
CRASH("generate() not implemented for [type]!")
/datum/asset/json/unregister()
SSassets.transport.unregister_asset("[name].json")