FabianK3andGitHub
36d267d83e
# Summary This PR has the goal of adding a base framework for tracking, saving and otherwise managing persistent data between rounds, written from scratch. Persistent data can be anything - Most expectedly noticeboards, dirt, papers, etc. _Technical details about the PR below are subject to change._ ## Estimated scope The first iteration of the persistence framework should include the functioning subsystem and the first implementation of a persistent type using said subsystem. - [x] Prepare database for persistent data - [x] Add new features to `/obj/` - [x] Logic to get and load persistent data into objects for the subsystem - [x] Internal logic for tracking and other - [x] Implement subsystem - [x] Basics, files, integration of subsystem - [x] Database queries for different operations - [x] Methods for registering and de-registering object tracking for `/obj` - [x] Round start logic - [x] Round end logic - [x] Implement first persistent type: Papers on notice boards - [ ] Code review - [x] Scope validation - [x] Testing - [x] Full changelog - Merge - [ ] Setup logging config for new subsystem - [x] Documentation for developers ([How to add new persistent types](https://github.com/Aurorastation/Aurora.3/wiki/Persistence)) ## Inner workings The new persistence subsystem has the following concept: At round start the subsystem reads existing data from the database and creates objects for them. During the round objects can be added to the tracking using a register function (or removed by a de-register function). At round end the subsystem goes over all it's actively tracked objects: Create new objects (that don't have a persistent ID from the database), update objects that changed during the round and expire objects that are no longer tracked. The objects in questions can register themselves, de-register themselves and decide themselves what content should be stored and handle how to set themselves up during the subsystems init phase. ## Information on the new table `id` - database generated ID for tracking persistent objects over multiple rounds, objects without an ID are considered new and created during the round. `author_ckey` - With the first implementation unused, but expected to be required in the future to allow any staff/moderation on persistent data. Nullable as persistent data also can be things like decals without an actual owner. `type` - Type of persistent data used for creating objects at round start. `created_at` - Statistical values. `expire_at` - Used for cleaning the database and setting limits for persistent data types. It's optional and allows permanent persistent data (command notice boards?). `content` - JSON formatted data containing the actual properties of an individual tracked object (e.g. what's written on a paper, what's its title). `x`, `y`, `z` - Coordinates taken and given from/to the object during save/load. Those can be null for future purposes. ## The goal on how to add new persistent data types `/obj` received two new methods which need to be overriden in the to-be added new persistent type: `persistence_get_content()` and `persistence_apply_content(content, x, y, z)`. The first method has the responsibility to provide all custom content of a new type to the subsystem in the form of an associated list (`["title" = "Hello, World!", "value" = 1337]. What does that mean? In the example of a piece of paper, the method needs to put the papers title and text content into a an associated list, that list will be saved by the persistence subsystem. The second method has the responsibility to apply the previously saved content when persistent data is initialized at the start of the round: The associated list needs to be read back and applied to the paper, additionally coordinates are provided for the object that have been taken during saving. Note that the coordinates might be null and could be ignored during setup. During the round, when a new object is created, like a paper, you need to call the subsystem and register a track: `SSpersistence.register_track(your_object_to_track, ckey (optional)`. The subsystem will be using the methods above when the round ends and saves the object - and loads it at the start of the round using the second method. The ckey needs to be given for all persistent that contains user generated content, e.g. papers. In case a persistent object gets removed, you need to call `SSpersistence.deregister_track(your_tracked_object)`. At the end of the round the object will be removed from the storage. `/obj` contains the variable (along some other technical vars) `persistance_initial_expiration_time_days`, which has a default value of 30 days, but can be safely overriden on a per-type basis. This value (in days) will be used for the persistent data entry expiration date when a new object of said type is stored in the database. ## PR description changelog - Added *The goal on how to add new persistent data types*. - Updates *Estimated scope* format. - Updates *Inner workings*. - Final iteration. *(Grammer and formatting not tracked manually)*
Compacted Migrations
To decrease the runtime of the migration unit test, the database migrations will be compacted into a single migration on a regular base. In order to do so, a new "migrate-VERSION" subfolder is created. The initial migration in these subfolders is always a migration with the current db-schema as of the current PR.
In addition the flayway.conf file in the root of the project is updated to use the new migration folder and create a new schema history table (that tracks the applied migrations).
How does this impact you?
If you set up a new database:
Make sure to use the latest migration folder, it will contain everything needed to create a "fresh" database.
If you have a existing database:
Update to the latest migration in the migration folder that you have used so far.
Then switch to the next migration folder (and a new schema version table)
You should use flyway with -baselineVersion="1" baseline instead of the usual migrate for the initial migration
As usual, always make sure that you have a backup and test it first on a non-production copy
Prerequisites
The server connects to a mysql-compatible server (mysql, mariadb, percona), so you'll need one of those with a database and user/password pair ready.
We use flyway to manage database migrations. To set up the database, you'll need to download flyway.
You'll also need some proficiency with the command line.
Attribution
Credit to Mloc from Baystation12 for the initial readme.
Creating migrations
As a coder, creating migrations is relatively easy. And they're a lot more flexible than just updating the initial schema would be.
First, figure out the changes you need to make. From table alteration and creation commands, to simply update and insert statements.
Write them into a .sql file in the SQL/migrate folder, in a valid order of execution. Name the file in the following format:
Vxxx__Description_goes_here.sql
Where xxx is the next version number from the last existing file (include the 0s), and the descrption is a short description for the migration, with spaces replaced by underscores.
Push this to your branch, and you're done!
Initial setup
In the root project directory, run:
path/to/flyway migrate -user=USER -password=PASSWORD -url=jdbc:mysql://HOST/DATABASE
Where USER is your mysql username, PASSWORD is your mysql password, HOST is the hostname of the mysql server and DATABASE is the database to use.
Migrating
Use the same command as above. Handy, isn't it?
Using a pre-flyway database
Note that this is not recommended! You may run into issues with some migrations, due to improper versioning. The best way to utilize this system is to set everything up on an empty schema. The next alternative is to make sure your database structure matches the V001 file within the migrate folder by manually modifying the structure to avoid dataloss, and then doing the steps described below.
If you're using a database since before we moved to flyway, it's a bit more involved to get migrations working.
In the root project directory, run:
path/to/flyway baseline -user=USER -password=PASSWORD -url=jdbc:mysql://HOST/DATABASE -baselineVersion=001 -baselineDescription="Initial schema"
From there, you can run migrations as normal.
Configuration file
Instead of putting -user, -password and -url in the command line every time you execute flyway, you can use a config file. Create it somewhere in the root of your project (we're calling it 'db.conf'):
flyway.url=jdbc:mysql://HOST/DATABASE
flyway.user=USER
flyway.password=PASSWORD
Now you can just run flyway migrate -configFile=db.conf, and the settings will be loaded from config.
Misc tables
We included a set of miscellanious tables in the misc folder. These are primarily used for debugging and are not meant to be pushed into production. As such, they're not included in the migration folder.
Ignoring or implementing them should not cause issues with the system.