mirror of
https://github.com/Aurorastation/Aurora.3.git
synced 2026-07-21 12:56:10 +01:00
Persistent data framework (#21055)
# Summary This PR has the goal of adding a base framework for tracking, saving and otherwise managing persistent data between rounds, written from scratch. Persistent data can be anything - Most expectedly noticeboards, dirt, papers, etc. _Technical details about the PR below are subject to change._ ## Estimated scope The first iteration of the persistence framework should include the functioning subsystem and the first implementation of a persistent type using said subsystem. - [x] Prepare database for persistent data - [x] Add new features to `/obj/` - [x] Logic to get and load persistent data into objects for the subsystem - [x] Internal logic for tracking and other - [x] Implement subsystem - [x] Basics, files, integration of subsystem - [x] Database queries for different operations - [x] Methods for registering and de-registering object tracking for `/obj` - [x] Round start logic - [x] Round end logic - [x] Implement first persistent type: Papers on notice boards - [ ] Code review - [x] Scope validation - [x] Testing - [x] Full changelog - Merge - [ ] Setup logging config for new subsystem - [x] Documentation for developers ([How to add new persistent types](https://github.com/Aurorastation/Aurora.3/wiki/Persistence)) ## Inner workings The new persistence subsystem has the following concept: At round start the subsystem reads existing data from the database and creates objects for them. During the round objects can be added to the tracking using a register function (or removed by a de-register function). At round end the subsystem goes over all it's actively tracked objects: Create new objects (that don't have a persistent ID from the database), update objects that changed during the round and expire objects that are no longer tracked. The objects in questions can register themselves, de-register themselves and decide themselves what content should be stored and handle how to set themselves up during the subsystems init phase. ## Information on the new table `id` - database generated ID for tracking persistent objects over multiple rounds, objects without an ID are considered new and created during the round. `author_ckey` - With the first implementation unused, but expected to be required in the future to allow any staff/moderation on persistent data. Nullable as persistent data also can be things like decals without an actual owner. `type` - Type of persistent data used for creating objects at round start. `created_at` - Statistical values. `expire_at` - Used for cleaning the database and setting limits for persistent data types. It's optional and allows permanent persistent data (command notice boards?). `content` - JSON formatted data containing the actual properties of an individual tracked object (e.g. what's written on a paper, what's its title). `x`, `y`, `z` - Coordinates taken and given from/to the object during save/load. Those can be null for future purposes. ## The goal on how to add new persistent data types `/obj` received two new methods which need to be overriden in the to-be added new persistent type: `persistence_get_content()` and `persistence_apply_content(content, x, y, z)`. The first method has the responsibility to provide all custom content of a new type to the subsystem in the form of an associated list (`["title" = "Hello, World!", "value" = 1337]. What does that mean? In the example of a piece of paper, the method needs to put the papers title and text content into a an associated list, that list will be saved by the persistence subsystem. The second method has the responsibility to apply the previously saved content when persistent data is initialized at the start of the round: The associated list needs to be read back and applied to the paper, additionally coordinates are provided for the object that have been taken during saving. Note that the coordinates might be null and could be ignored during setup. During the round, when a new object is created, like a paper, you need to call the subsystem and register a track: `SSpersistence.register_track(your_object_to_track, ckey (optional)`. The subsystem will be using the methods above when the round ends and saves the object - and loads it at the start of the round using the second method. The ckey needs to be given for all persistent that contains user generated content, e.g. papers. In case a persistent object gets removed, you need to call `SSpersistence.deregister_track(your_tracked_object)`. At the end of the round the object will be removed from the storage. `/obj` contains the variable (along some other technical vars) `persistance_initial_expiration_time_days`, which has a default value of 30 days, but can be safely overriden on a per-type basis. This value (in days) will be used for the persistent data entry expiration date when a new object of said type is stored in the database. ## PR description changelog - Added *The goal on how to add new persistent data types*. - Updates *Estimated scope* format. - Updates *Inner workings*. - Final iteration. *(Grammer and formatting not tracked manually)*
This commit is contained in:
FabianK3
@@ -62,6 +62,7 @@ GLOBAL_LIST_EMPTY(gamemode_cache)
|
||||
"log_subsystems_ghostroles" = TRUE, // Ghost Roles
|
||||
"log_subsystems_law" = TRUE, // Law
|
||||
"log_subsystems_cargo" = TRUE, // Cargo
|
||||
"log_subsystems_persistence" = TRUE, // Persistence
|
||||
"log_subsystems_documents" = TRUE, // Documents
|
||||
"log_subsystems_fail2topic" = TRUE, // Fail2Topic
|
||||
"log_subsystems_mapfinalization" = TRUE, // Map Finalization
|
||||
@@ -120,6 +121,7 @@ GLOBAL_LIST_EMPTY(gamemode_cache)
|
||||
"world_subsystems_ghostroles_log" = "subsystems/ghostroles.log",
|
||||
"world_subsystems_law_log" = "subsystems/law.log",
|
||||
"world_subsystems_cargo_log" = "subsystems/cargo.log",
|
||||
"world_subsystems_persistence_log" = "subsystems/persistence.log",
|
||||
"world_subsystems_documents_log" = "subsystems/documents.log",
|
||||
"world_subsystems_fail2topic_log" = "subsystems/fail2topic.log",
|
||||
"world_subsystems_mapfinalization_log" = "subsystems/mapfinalization.log",
|
||||
|
||||
@@ -0,0 +1,310 @@
|
||||
SUBSYSTEM_DEF(persistence)
|
||||
name = "Persistence"
|
||||
init_order = INIT_ORDER_PERSISTENCE
|
||||
flags = SS_NO_FIRE
|
||||
|
||||
/*#############################################
|
||||
Internal methods
|
||||
#############################################*/
|
||||
|
||||
/**
|
||||
* Initialization of the persistence subsystem. Initialization includes loading all persistent data and spawning the related objects.
|
||||
*/
|
||||
/datum/controller/subsystem/persistence/Initialize()
|
||||
. = ..()
|
||||
try
|
||||
if(!GLOB.config.sql_enabled)
|
||||
log_subsystem_persistence("SQL configuration not enabled! Persistence subsystem requires SQL.")
|
||||
return SS_INIT_SUCCESS
|
||||
|
||||
GLOB.persistence_register = list()
|
||||
|
||||
if(!SSdbcore.Connect())
|
||||
log_subsystem_persistence("SQL ERROR during persistence subsystem init. Failed to connect.")
|
||||
return SS_INIT_FAILURE
|
||||
else
|
||||
// Delete all persistent objects in the database that have expired and have passed the cleanup grace period (PERSISTENT_EXPIRATION_CLEANUP_DELAY_DAYS)
|
||||
database_clean_entries()
|
||||
|
||||
// Retrieve all persistent data that is not expired
|
||||
var/list/persistent_data = database_get_active_entries()
|
||||
log_subsystem_persistence("Init: Retrieved [persistent_data.len] entries for instancing this round.")
|
||||
|
||||
// Instantiate all remaining entries based of their type
|
||||
// Assign persistence related vars found in /obj, apply content and add to live tracking list.
|
||||
for (var/data in persistent_data)
|
||||
CHECK_TICK
|
||||
var/typepath = text2path(data["type"])
|
||||
if (!ispath(typepath)) // Type checking
|
||||
continue
|
||||
var/obj/instance = new typepath()
|
||||
instance.persistence_track_id = data["id"]
|
||||
track_apply_content(instance, data["content"], data["x"], data["y"], data["z"])
|
||||
register_track(instance, data["author_ckey"])
|
||||
|
||||
return SS_INIT_SUCCESS
|
||||
catch(var/exception/e)
|
||||
log_subsystem_persistence("Panic: Exception during subsystem initialize: [e]")
|
||||
return SS_INIT_FAILURE
|
||||
|
||||
/**
|
||||
* Shutdown of the persistence subsystem. Adds new persistent objects, removes no longer existing persistent objects and updates changed persistent objects in the database.
|
||||
*/
|
||||
/datum/controller/subsystem/persistence/Shutdown()
|
||||
try
|
||||
if(!GLOB.config.sql_enabled)
|
||||
log_subsystem_persistence("SQL configuration not enabled! Panic - Cannot save current round track changes!")
|
||||
return
|
||||
|
||||
// Subsystem shutdown:
|
||||
// Create new persistent records for objects that have been created in the round
|
||||
// Update tracked objects that have an ID (already existing from previous rounds)
|
||||
// Delete persistent records that no longer exist in the registry (removed during the round)
|
||||
|
||||
var/created = 0
|
||||
var/updated = 0
|
||||
var/expired = 0
|
||||
|
||||
// Get already stored data before saving new tracks so we can compare what has been updated or removed during the round.
|
||||
var/list/existing_data = database_get_active_entries()
|
||||
|
||||
for (var/obj/track in GLOB.persistence_register)
|
||||
CHECK_TICK
|
||||
if (track.persistence_track_id == 0)
|
||||
// Tracked object has no ID meaning it is new, create a new persistent record for it
|
||||
database_add_entry(track)
|
||||
created++
|
||||
|
||||
// Find tracks that have been removed during the round by trying to find the track by database ID
|
||||
// If we find the track, we need to check if it requires an update instead
|
||||
for (var/record in existing_data)
|
||||
var/found = FALSE
|
||||
for (var/obj/track in GLOB.persistence_register)
|
||||
CHECK_TICK
|
||||
if (record["id"] == track.persistence_track_id)
|
||||
// A track with the same ID has been found in the register, it still exists, check if we need to update it instead
|
||||
found = TRUE // Prevent expiration of track
|
||||
var/changed = FALSE
|
||||
var/turf/T = get_turf(track)
|
||||
if (T && T.x != record["x"])
|
||||
changed = TRUE
|
||||
else if (T && T.y != record["y"])
|
||||
changed = TRUE
|
||||
else if (T && T.z != record["z"])
|
||||
changed = TRUE
|
||||
else if (track_get_content(track) != record["content"])
|
||||
changed = TRUE
|
||||
if (changed)
|
||||
database_update_entry(track)
|
||||
updated++
|
||||
break // Track found (and perhaps updated), break off loop search as it won't need to be deleted anyways
|
||||
if (!found)
|
||||
// No track with the same ID has been found in the register, remove it from the database (expire)
|
||||
database_expire_entry(record["id"])
|
||||
expired++
|
||||
|
||||
log_subsystem_persistence("Shutdown: Tried to create [created], update [updated] and expire [expired] tracks.")
|
||||
catch(var/exception/e)
|
||||
log_subsystem_persistence("Panic: Exception during subsystem shutdown: [e]")
|
||||
return
|
||||
|
||||
/**
|
||||
* Generates StatEntry. Returns information about currently tracked objects.
|
||||
*/
|
||||
/datum/controller/subsystem/persistence/stat_entry(msg)
|
||||
msg = ("Global register tracks: [GLOB.persistence_register.len]")
|
||||
return ..()
|
||||
|
||||
/**
|
||||
* Safely get JSON persistent content of track.
|
||||
* RETURN: JSON formatted content of track or null if an exception occured.
|
||||
*/
|
||||
/datum/controller/subsystem/persistence/proc/track_get_content(var/obj/track)
|
||||
var/result = null
|
||||
try
|
||||
result = json_encode(track.persistence_get_content())
|
||||
catch(var/exception/e)
|
||||
log_subsystem_persistence("Track: Failed to get/encode track content: [e]")
|
||||
return result
|
||||
|
||||
/**
|
||||
* Safely apply persistent content to track.
|
||||
* PARAMS:
|
||||
* track = Object to apply content to.
|
||||
* json = Custom persistent content JSON to be applied.
|
||||
* x,y,z = x-y-z coordinates of object, can be null.
|
||||
*/
|
||||
/datum/controller/subsystem/persistence/proc/track_apply_content(var/obj/track, var/json, var/x, var/y, var/z)
|
||||
try
|
||||
track.persistence_apply_content(json_decode(json), x, y, z)
|
||||
catch(var/exception/e)
|
||||
log_subsystem_persistence("Track: Failed to apply/decode track content: [e]")
|
||||
|
||||
/**
|
||||
* Run cleanup on the persistence entries in the database.
|
||||
* Cleanup includes all entries that have expired and have passed the clean up grace period (PERSISTENT_EXPIRATION_CLEANUP_DELAY_DAYS).
|
||||
*/
|
||||
/datum/controller/subsystem/persistence/proc/database_clean_entries()
|
||||
if(!SSdbcore.Connect())
|
||||
log_subsystem_persistence("SQL ERROR during persistence database_clean_entries. Failed to connect.")
|
||||
else
|
||||
var/datum/db_query/cleanup_query = SSdbcore.NewQuery(
|
||||
"DELETE FROM ss13_persistent_data WHERE DATE_ADD(expires_at, INTERVAL :grace_period_days DAY) <= NOW()",
|
||||
list("grace_period_days" = PERSISTENT_EXPIRATION_CLEANUP_DELAY_DAYS)
|
||||
)
|
||||
|
||||
cleanup_query.SetFailCallback(CALLBACK(PROC_REF(database_clean_entries_callback_failure)))
|
||||
cleanup_query.SetSuccessCallback(CALLBACK(GLOBAL_PROC, GLOBAL_PROC_REF(qdel)))
|
||||
cleanup_query.ExecuteNoSleep(TRUE)
|
||||
|
||||
/datum/controller/subsystem/persistence/proc/database_clean_entries_callback_failure(var/datum/db_query/cleanup_query)
|
||||
if (cleanup_query.ErrorMsg())
|
||||
log_subsystem_persistence("SQL ERROR during persistence database_clean_entries. " + cleanup_query.ErrorMsg())
|
||||
qdel(cleanup_query)
|
||||
|
||||
/**
|
||||
* Retrieve persistent data entries that haven't expired.
|
||||
* RETURN: List of JSON, with ID, author_ckey, type, content, x, y, z
|
||||
*/
|
||||
/datum/controller/subsystem/persistence/proc/database_get_active_entries()
|
||||
if(!SSdbcore.Connect())
|
||||
log_subsystem_persistence("SQL ERROR during persistence database_get_active_entries. Failed to connect.")
|
||||
else
|
||||
var/datum/db_query/get_query = SSdbcore.NewQuery(
|
||||
"SELECT id, author_ckey, type, content, x, y, z FROM ss13_persistent_data WHERE NOW() < expires_at"
|
||||
)
|
||||
get_query.Execute()
|
||||
|
||||
var/list/results = list()
|
||||
if (get_query.ErrorMsg())
|
||||
log_subsystem_persistence("SQL ERROR during persistence database_get_active_entries. " + get_query.ErrorMsg())
|
||||
return
|
||||
else
|
||||
while (get_query.NextRow())
|
||||
CHECK_TICK
|
||||
var/list/entry = list(
|
||||
"id" = get_query.item[1],
|
||||
"author_ckey" = get_query.item[2],
|
||||
"type" = get_query.item[3],
|
||||
"content" = get_query.item[4],
|
||||
"x" = get_query.item[5],
|
||||
"y" = get_query.item[6],
|
||||
"z" = get_query.item[7]
|
||||
)
|
||||
results += list(entry)
|
||||
qdel(get_query)
|
||||
return results
|
||||
|
||||
/**
|
||||
* Adds a persistent data record to the database.
|
||||
*/
|
||||
/datum/controller/subsystem/persistence/proc/database_add_entry(var/obj/track)
|
||||
if(!SSdbcore.Connect())
|
||||
log_subsystem_persistence("SQL ERROR during persistence database_add_entry. Failed to connect.")
|
||||
else
|
||||
var/content = track_get_content(track)
|
||||
if (!content)
|
||||
return
|
||||
|
||||
var/turf/T = get_turf(track)
|
||||
if(!T || !is_station_level(T.z)) // The persistence system only supports objects from the main map levels for multiple reasons, e.g. Z level value, mapping support
|
||||
return
|
||||
|
||||
var/datum/db_query/insert_query = SSdbcore.NewQuery(
|
||||
"INSERT INTO ss13_persistent_data (author_ckey, type, created_at, expires_at, content, x, y, z) \
|
||||
VALUES (:author_ckey, :type, NOW(), DATE_ADD(NOW(), INTERVAL :expire_in_days DAY), :content, :x, :y, :z)",
|
||||
list(
|
||||
"author_ckey" = track.persistence_author_ckey,
|
||||
"type" = "[track.type]",
|
||||
"expire_in_days" = track.persistance_expiration_time_days,
|
||||
"content" = content,
|
||||
"x" = T.x,
|
||||
"y" = T.y,
|
||||
"z" = T.z
|
||||
)
|
||||
)
|
||||
insert_query.Execute()
|
||||
|
||||
if (insert_query.ErrorMsg())
|
||||
log_subsystem_persistence("SQL ERROR during persistence database_add_entry. " + insert_query.ErrorMsg())
|
||||
qdel(insert_query)
|
||||
|
||||
/**
|
||||
* Updates a persistent data record in the database.
|
||||
*/
|
||||
/datum/controller/subsystem/persistence/proc/database_update_entry(var/obj/track)
|
||||
if(!SSdbcore.Connect())
|
||||
log_subsystem_persistence("SQL ERROR during persistence database_update_entry. Failed to connect.")
|
||||
else
|
||||
var/content = track_get_content(track)
|
||||
if (!content)
|
||||
return
|
||||
|
||||
var/turf/T = get_turf(track)
|
||||
if(!T || !is_station_level(T.z)) // The persistence system only supports objects from the main map levels for multiple reasons, e.g. Z level value, mapping support
|
||||
return
|
||||
|
||||
var/datum/db_query/update_query = SSdbcore.NewQuery(
|
||||
"UPDATE ss13_persistent_data SET author_ckey=:author_ckey, expires_at=DATE_ADD(NOW(), INTERVAL :expire_in_days DAY), content=:content, x=:x, y=:y, z=:z WHERE id = :id",
|
||||
list(
|
||||
"author_ckey" = track.persistence_author_ckey,
|
||||
"expire_in_days" = track.persistance_expiration_time_days,
|
||||
"content" = content,
|
||||
"x" = T.x,
|
||||
"y" = T.y,
|
||||
"z" = T.z,
|
||||
"id" = track.persistence_track_id
|
||||
)
|
||||
)
|
||||
update_query.Execute()
|
||||
|
||||
if (update_query.ErrorMsg())
|
||||
log_subsystem_persistence("SQL ERROR during persistence database_update_entry. " + update_query.ErrorMsg())
|
||||
qdel(update_query)
|
||||
|
||||
/**
|
||||
* Expire a persistent data record in the database by setting it's expiration date to now.
|
||||
*/
|
||||
/datum/controller/subsystem/persistence/proc/database_expire_entry(var/track_id)
|
||||
if(!SSdbcore.Connect())
|
||||
log_subsystem_persistence("SQL ERROR during persistence database_expire_entry. Failed to connect.")
|
||||
else
|
||||
var/datum/db_query/expire_query = SSdbcore.NewQuery(
|
||||
"UPDATE ss13_persistent_data SET expires_at=NOW() WHERE id = :id",
|
||||
list("id" = track_id)
|
||||
)
|
||||
expire_query.Execute()
|
||||
|
||||
if (expire_query.ErrorMsg())
|
||||
log_subsystem_persistence("SQL ERROR during persistence database_expire_entry. " + expire_query.ErrorMsg())
|
||||
qdel(expire_query)
|
||||
|
||||
/*#############################################
|
||||
Public methods
|
||||
#############################################*/
|
||||
|
||||
/**
|
||||
* Adds the given object to the list of tracked objects. At shutdown the tracked object will be either created or updated in the database.
|
||||
* The ckey is an optional argument and is used for tracking user generated content by adding an author to the persistent data.
|
||||
*/
|
||||
/datum/controller/subsystem/persistence/proc/register_track(var/obj/new_track, var/ckey)
|
||||
if(new_track.persistence_track_active) // Prevent multiple registers per object and removes the need to check the register if it's already in there
|
||||
return
|
||||
|
||||
var/turf/T = get_turf(new_track)
|
||||
if(!T || !is_station_level(T.z)) // The persistence system only supports objects from the main map levels for multiple reasons, e.g. Z level value, mapping support
|
||||
return
|
||||
|
||||
new_track.persistence_track_active = TRUE
|
||||
new_track.persistence_author_ckey = ckey
|
||||
GLOB.persistence_register += new_track
|
||||
|
||||
/**
|
||||
* Removes the given object from the list of tracked objects. At shutdown the tracked object will be remove from the database.
|
||||
*/
|
||||
/datum/controller/subsystem/persistence/proc/deregister_track(var/obj/old_track)
|
||||
if(!old_track.persistence_track_active) // Prevent multiple deregisters per object and removes the need to check the register if it's not in there
|
||||
return
|
||||
|
||||
old_track.persistence_track_active = FALSE
|
||||
GLOB.persistence_register -= old_track
|
||||
Reference in New Issue
Block a user