tgstation-server/docs/Architecture.dox

/*!
@page architecture Server Architecture

@tableofcontents

@image html ArchitectureOverview.png "Architecture Overview"

@section arch_intro Introduction

This is meant to be a brief overview of the TGS4 architecture to give new coders direction on where to code and the curious some insight to their questions. Given that this document is seperate from the authorative code it may fall out of date. For clairity, please contact project maintainers.

@section arch_hwatchdog Host Watchdog

The host watchdog process monitors the actual server application. It exists mainly to facilitate the live update system built into it and also serves as a restart mechanism if requested. This component is generally not present in active development scenarios due to debugging overhead.

This consists of two parts a runner and the watchdog library. The library is the Tgstation.Server.Host.Watchdog project and the runners are the Tgstation.Server.Host.Console .NET Core project and the Tgstation.Server.Host.Service .NET Framework Windows service project

@section arch_main Main Server

This is a second process spawned by the Host Watchdog which facilitates the vast majority of the code. This is the Tgstation.Server.Host project which is fundamentally an ASP.NET Core MVC web application.

@subsection arch_setup Server Initialization

The server's entrypoint is in the @ref Tgstation.Server.Host.Program class. This class mainly determines if the Host watchdog is present and creates and runs the @ref Tgstation.Server.Host.Server class. That class then builds an ASP.NET Core web host using the @ref Tgstation.Server.Host.Core.Application class.

The @ref Tgstation.Server.Host.Core.Application class has two methods called by the framework. First the @ref Tgstation.Server.Host.Core.Application.ConfigureServices method sets up dependency injection of interfaces for Controllers, the @ref Tgstation.Server.Host.Models.DatabaseContext, and the component factories of the server. The framework handles constructing these things once the application starts. Configuration is loaded from the appropriate appSettings.json into the @ref Tgstation.Server.Host.Configuration classes for injection as well. Then @ref Tgstation.Server.Host.Core.Application.Configure method is run which sets up the web request pipeline which currently has the following stack of handlers:

- Catch any exceptions and respond with 500 and detailed HTML error page
- Respond with 503 if the application is still starting or shutting down
- Authenticate the JWT in Authentication header if present and run @ref Tgstation.Server.Host.Controllers.ApiController on success
- Catch database exceptions and convert to 409 responses with the exception's @ref Tgstation.Server.Api.Models.ErrorMessage
- Check @ref Tgstation.Server.Host.Controllers for correct controller and run the action and use it's response. 
	- If not properly authenticated beforehand and action has a @ref Tgstation.Server.Host.Controllers.TgsAuthorizeAttribute return 401
	- If not properly authorized beforehand according to the parameters of the action's @ref Tgstation.Server.Host.Controllers.TgsAuthorizeAttribute (if present) return 403
	- If requested action does not exist return 404

@subsubsection arch_instinit Instance Manager Initialization

Once the web host starts, the @ref Tgstation.Server.Host.Components.InstanceManager.StartAsync function is called (due to being registered as a IHostedService in @ref Tgstation.Server.Host.Core.Application) this is the only StartAsync implementation that should be called by the framework, others should be called from this to maintain a cohesive initialization order.

The first thing this function does is call @ref Tgstation.Server.Host.Models.DatabaseContext.Initialize which ensures the database is migrated, seeded, and ready to go. Then the @ref Tgstation.Server.Host.Core.JobManager is started, which cleans up any jobs that are considered "still running" in the database. Finally all instances configured to be online are created in parallel (See @ref arch_instance for onlining process) and the @ref Tgstation.Server.Host.Core.Application is signalled to stop blocking requests with 503 responses before they are processed.

@section arch_db Database and Context

The database is exposed as a series of DbSet<T> objects through @ref Tgstation.Server.Host.Models.IDatabaseContext . Queries are performed via async LINQ expressions. Inserts, updates, and deletes are done via modifiying the DbSet<T>s and then calling @ref Tgstation.Server.Host.Models.IDatabaseContext.Save . Do some reading on Entity Framework Core for a deeper understanding.

@section arch_controllers Controllers

The webserver operates in an MVC style. All requests are routed through the @ref Tgstation.Server.Host.Controllers . If a route doesn't exist as an action in a controller, a 404 response will be returned. Controllers interact with components via injecting the @ref Tgstation.Server.Host.Components.IInstanceManager interface, access the database with the @ref Tgstation.Server.Host.Controllers.ApiController.DatabaseContext property, and start jobs by injecting the @ref Tgstation.Server.Host.Core.IJobManager interface.

@section arch_security Security

The authentication process begins in @ref Tgstation.Server.Host.Controllers.HomeController.CreateToken . This is where users log in. They must supply their username and password via correct @ref Tgstation.Server.Api.ApiHeaders . The server first attempts to use these credentials to login to the system. If that succeeds it checks if the system user's UID is registered in the database. Failing either of the previous two, it tries to match the username and password to an entry in the database. If either of these methods succeeds the user is considered authenticated and a token is generated and sent back to the user. If the user is a system user, the context of their login is kept for the amount of time until their token expires + 1 minute.

The password hashing used for database users is the standard provided by ASP.Net Core. It utilizes PBKDF2 with HMAC-SHA256, 128-bit salt, 256-bit subkey, with 10000 iterations. Read about it here: https://andrewlock.net/exploring-the-asp-net-core-identity-passwordhasher/

When this token is supplied in the `Authorization` header of a subsequent request, it is first cryptographically validated that it was sent by the current server. The token contain's the user's ID, and, using it, the user's info is retrieved from the database and put into an @ref Tgstation.Server.Host.Security.IAuthenticationContext

Nearly all exposed controller actions are decorated with a @ref Tgstation.Server.Host.Controllers.TgsAuthorizeAttribute . This attribute does 2 things. 1. It ensures the @ref Tgstation.Server.Host.Security.IAuthenticationContext is valid for the request before running the action. 2. If it contains a permission flag specification, it will 403 the request if the user doesn't have one of the listed permissions.

@section arch_jobs Jobs

Long running operations create @ref Tgstation.Server.Host.Models.Job objects which represent information about long running tasks. These objects can be queried to find out who started them, if they've been completed, canceled, who cancelled them, their error message if any, and get their progress percentage in some cases. The job will be created and supplied by the request that started it, but active/all jobs may also be queried.

@section arch_instance Instances

Instances exist in two forms: Their database metadata and their actual class. The class only exists if the instance is set to be @ref Tgstation.Server.Api.Models.Instance.Online . This is where all the actual server management code lives. A single instance is made up of individual components that work with each other through their intefaces

@subsection arch_ifactory Instance Factory

This is responsible for creating the components and weaving them into the final @ref Tgstation.Server.Host.Components.Instance. This happens automatically at server startup if an instance is configured to be online

@section arch_repository Repository Manager

The @ref Tgstation.Server.Host.Components.Repository.IRepositoryManager is the gatekeeper for cloning and accessing a @ref Tgstation.Server.Host.Components.Repository.IRepository . Only one instance of a repository can be in use at a time (due to the single-threaded nature of libgit2), so the repository manager contains a semaphore wait queue which hands out the repository to only one client at a time. All repository operations (aside from cloning and deleting) are performed by the actual repository object. This includes fetching, hard resets, checkouts, synchronizing, etc. Most state put into the @ref Tgstation.Server.Api.Models.Repository object is read directly from libgit2, exceptions being credentials and boolean settings.

@section arch_byond Byond

The BYOND installation setup is largely decoupled from the database. When a byond version is downloaded and installed by the @ref Tgstation.Server.Host.Components.Byond.IByondManager it is extracted to a directory titled with it's version in the `BYOND` folder acompanied by a text document stating which version it is. Platform specific installation steps are handled by specific implementations of @ref Tgstation.Server.Host.Components.Byond.IByondInstaller . When it comes time to use an executable, the manager provides a @ref Tgstation.Server.Host.Components.Byond.IByondExecutableLock which contains absolute paths to the DreamMaker and DreamDaemon executables

@section arch_deployment Compiler and Deployment

The compilation process is a distinct series of steps:

1. Retrieve necessary information from the database in @ref Tgstation.Server.Host.Components.IInstance.CompileProcess
2. Choose a uniquely named folder in the `Game` directory for deployment
3. Acquire a @ref Tgstation.Server.Host.Components.Byond.IByondExecutableLock
4. Build the initial @ref Tgstation.Server.Host.Models.CompileJob object from available data (Byond version, Revision, directory, etc)
5. Announce the deployment through the chat bot system
6. Create and copy the repository to the `<target folder>/A` directory
7. Run the PreCompile hook
8. Auto detect or check if the configured .dme is present
9. Copy and apply static code modifications to the environment
10. Run DreamMaker on the .dme
11. Start a DreamDaemon instance to validate the DMAPI
12. Run the PostCompile hook
13. Copy `<target folder>/A` to `<target folder>/B`
14. Symlink all `GameStaticFiles` to both the A and B directories
15. Commit the @ref Tgstation.Server.Host.Models.CompileJob to the database

If any of the above steps fail, the target directory is deleted and the deployment is considered a bust. If all went well, after the @ref Tgstation.Server.Host.Models.Job completes the new CompileJob is loaded into the instance's @ref Tgstation.Server.Host.Components.Compiler.IDmbFactory . 

The DmbFactory is where the @ref arch_watchdog gets the @ref Tgstation.Server.Host.Components.Compiler.IDmbProvider instances to run. Each CompileJob loaded into it is given a lock count. The latest CompileJob holds 1 lock and every DreamDaemon instance running that CompileJob holds another. Loading a new CompileJob releases the initial lock, and when all other locks are released the CompileJob's directory is deleted. Any directories in the `Game` folder not in use are also deleted when the Instance starts. 

@section arch_chat Chat Bot System

@section arch_static Static File Management

@section arch_watchdog Watchdog

@subsection arch_comms Communication

@section arch_update Host Update Process

*/