[MIRROR] Biddle Verbs: Queues the Most Expensive Verbs for the Next Tick if the Server Is Overloaded [MDB IGNORE] (#15329)

* Biddle Verbs: Queues the Most Expensive Verbs for the Next Tick if the Server Is Overloaded (#65589)

This pr goes through: /client/Click(), /client/Topic(), /mob/living/verb/resist(), /mob/verb/quick_equip(), /mob/verb/examinate(), and /mob/verb/mode() and makes them queue their functionality to a subsystem to execute in the next tick if the server is overloaded. To do this a new subsystem is made to handle most verbs called SSverb_manager, if the server is overloaded the verb queues itself in the subsystem and returns, then near the start of the next tick that verb is resumed with the provided callback. The verbs are called directly after SSinput, and the subsystem does not yield until its queue is completely finished.

The exception are clicks from player input since they are extremely important for the feeling of responsiveness. I considered not queuing them but theyre too expensive not to, suffering from a death of a thousand cuts performance wise from many many things in the process adding up. Instead clicks are executed at the very start of the next tick, as the first action that SSinput completes, before player movement is processed even.

A few months ago, before I died I was trying to figure out why games at midpop (40-50 people) had non zero and consistent time dilation without maptick being consistently above 28% (which is when the MC stops yielding for maptick if its overloaded). I found it out, started working on this pr, then promptly died. luckily im a bit less dead now

the current MC has a problem: the cost of verbs is completely and totally invisible to it, it cannot account for them. Why is this bad? because verbs are the last thing to execute in the tick, after the MC and SendMaps have finished executing.
tick diagram2
If the MC is overloaded and uses 100% of the time it allots itself this means that if SendMaps uses the amount its expected to take, verbs have at most 2% of the tick to execute in before they are overtiming and thus delaying the start of the next tick. This is bad, and im 99% sure this is the majority of our overtime.

Take Click() for example. Click isnt listed as a verb but since its called as a result of client commands its executed at the end of the tick like other verbs. in this random 80 pop sybil round profile i had saved on my computer sybil 80 pop (2).txt /client/Click() has an overtime of only 1.8 seconds, which isnt that bad. however it has a self cpu of 2.5 seconds meaning 1.8/2.5 = 72% of its time is overtiming, and it also is calling 80.2 seconds worth of total cpu, which means that more than 57.7 seconds of overtime is attributed to just /client/Click() executing at the very end of a tick. the reason why this isnt obvious is just because the verbs themselves typically dont have high enough self cpu to get high enough on the rankings of overtiming procs to be noticed, all of their overtime is distributed among a ton of procs they call in the chain.

Since i cant guarantee the MC resumes at the very start of the next tick due to other sleeping procs almost always resuming first: I time the duration between clicks being queued up for the next tick and when theyre actually executed. if it exceeds 20 milliseconds of added latency (less than one tenth the average human reaction time) clicks will execute immediately instead of queuing, this should make instances where a player can notice the added latency a vanishingly small minority of cases. still, this should be tm'd

* Biddle Verbs: Queues the Most Expensive Verbs for the Next Tick if the Server Is Overloaded

Co-authored-by: Kylerace <kylerlumpkin1@gmail.com>

.github/guides/TICK_ORDER.md

@@ -0,0 +1,21 @@
+The byond tick proceeds as follows:
+1. procs sleeping via walk() are resumed (i dont know why these are first)
+
+2. normal sleeping procs are resumed, in the order they went to sleep in the first place, this is where the MC wakes up and processes subsystems. a consequence of this is that the MC almost never resumes before other sleeping procs, because it only goes to sleep for 1 tick 99% of the time, and 99% of procs either go to sleep for less time than the MC (which guarantees that they entered the sleep queue earlier when its time to wake up) and/or were called synchronously from the MC's execution, almost all of the time the MC is the last sleeping proc to resume in any given tick. This is good because it means the MC can account for the cost of previous resuming procs in the tick, and minimizes overtime.
+
+3. control is passed to byond after all of our code's procs stop execution for this tick
+
+4. a few small things happen in byond internals
+
+5. SendMaps is called for this tick, which processes the game state for all clients connected to the game and handles sending them changes
+in appearances within their view range. This is expensive and takes up a significant portion of our tick, about 0.45% per connected player
+as of 3/20/2022. meaning that with 50 players, 22.5% of our tick is being used up by just SendMaps, after all of our code has stopped executing. Thats only the average across all rounds, for most highpop rounds it can look like 0.6% of the tick per player, which is 30% for 50 players.
+
+6. After SendMaps ends, client verbs sent to the server are executed, and its the last major step before the next tick begins.
+During the course of the tick, a client can send a command to the server saying that they have executed any verb. The actual code defined
+for that /verb/name() proc isnt executed until this point, and the way the MC is designed makes this especially likely to make verbs
+"overrun" the bounds of the tick they executed in, stopping the other tick from starting and thus delaying the MC firing in that tick.
+
+The master controller can derive how much of the tick was used in: procs executing before it woke up (because of world.tick_usage), and SendMaps (because of world.map_cpu, since this is a running average you cant derive the tick spent on maptick on any particular tick). It cannot derive how much of the tick was used for sleeping procs resuming after the MC ran, or for verbs executing after SendMaps. 
+
+It is for these reasons why you should heavily limit processing done in verbs, while procs resuming after the MC are rare, verbs are not, and are much more likely to cause overtime since theyre literally at the end of the tick. If you make a verb, try to offload any expensive work to the beginning of the next tick via a verb management subsystem.

code/__DEFINES/MC.dm

@@ -17,6 +17,15 @@
 #define MC_AVG_FAST_UP_SLOW_DOWN(average, current) (average > current ? MC_AVERAGE_SLOW(average, current) : MC_AVERAGE_FAST(average, current))
 #define MC_AVG_SLOW_UP_FAST_DOWN(average, current) (average < current ? MC_AVERAGE_SLOW(average, current) : MC_AVERAGE_FAST(average, current))
 
+///creates a running average of "things elapsed" per time period when you need to count via a smaller time period.
+///eg you want an average number of things happening per second but you measure the event every tick (50 milliseconds).
+///make sure both time intervals are in the same units. doesnt work if current_duration > total_duration or if total_duration == 0
+#define MC_AVG_OVER_TIME(average, current, total_duration, current_duration) (((total_duration - current_duration) / (total_duration)) * (average) + (current))
+
+#define MC_AVG_MINUTES(average, current, current_duration) (MC_AVG_OVER_TIME(average, current, 1 MINUTES, current_duration))
+
+#define MC_AVG_SECONDS(average, current, current_duration) (MC_AVG_OVER_TIME(average, current, 1 SECONDS, current_duration))
+
 #define NEW_SS_GLOBAL(varname) if(varname != src){if(istype(varname)){Recover();qdel(varname);}varname = src;}
 
 #define START_PROCESSING(Processor, Datum) if (!(Datum.datum_flags & DF_ISPROCESSING)) {Datum.datum_flags |= DF_ISPROCESSING;Processor.processing += Datum}
@@ -110,3 +119,11 @@
 }\
 /datum/controller/subsystem/fluids/##X/fire() {..() /*just so it shows up on the profiler*/} \
 /datum/controller/subsystem/fluids/##X
+
+#define VERB_MANAGER_SUBSYSTEM_DEF(X) GLOBAL_REAL(SS##X, /datum/controller/subsystem/verb_manager/##X);\
+/datum/controller/subsystem/verb_manager/##X/New(){\
+	NEW_SS_GLOBAL(SS##X);\
+	PreInit();\
+}\
+/datum/controller/subsystem/verb_manager/##X/fire() {..() /*just so it shows up on the profiler*/} \
+/datum/controller/subsystem/verb_manager/##X

code/__DEFINES/callbacks.dm

@@ -2,3 +2,5 @@
 /// A shorthand for the callback datum, [documented here](datum/callback.html)
 #define CALLBACK new /datum/callback
 #define INVOKE_ASYNC world.ImmediateInvokeAsync
+/// like CALLBACK but specifically for verb callbacks
+#define VERB_CALLBACK new /datum/callback/verb_callback

code/__DEFINES/input.dm

@@ -0,0 +1,2 @@
+///if the running average click latency is above this amount then clicks will never queue and will execute immediately
+#define MAXIMUM_CLICK_LATENCY (20 MILLISECONDS)

code/__DEFINES/subsystems.dm

@@ -208,6 +208,7 @@
 #define FIRE_PRIORITY_TIMER 700
 #define FIRE_PRIORITY_SOUND_LOOPS 800
 #define FIRE_PRIORITY_SPEECH_CONTROLLER 900
+#define FIRE_PRIORITY_DELAYED_VERBS 950
 #define FIRE_PRIORITY_INPUT 1000 // This must always always be the max highest priority. Player input must never be lost.
 
 

code/__DEFINES/time.dm

@@ -46,6 +46,10 @@ When using time2text(), please use "DDD" to find the weekday. Refrain from using
 #define SATURDAY "Sat"
 #define SUNDAY "Sun"
 
+#define MILLISECONDS *0.01
+
+#define DECISECONDS *1 //the base unit all of these defines are scaled by, because byond uses that as a unit of measurement for some fucking reason
+
 #define SECONDS *10
 
 #define MINUTES SECONDS*60
@@ -54,8 +58,6 @@ When using time2text(), please use "DDD" to find the weekday. Refrain from using
 
 #define TICKS *world.tick_lag
 
-#define MILLISECONDS * 0.01
-
 #define DS2TICKS(DS) ((DS)/world.tick_lag)
 
 #define TICKS2DS(T) ((T) TICKS)

code/__DEFINES/verb_manager.dm

@@ -0,0 +1,36 @@
+/**
+ * verb queuing thresholds. remember that since verbs execute after SendMaps the player wont see the effects of the verbs on the game world
+ * until SendMaps executes next tick, and then when that later update reaches them. thus most player input has a minimum latency of world.tick_lag + player ping.
+ * however thats only for the visual effect of player input, when a verb processes the actual latency of game state changes or semantic latency is effectively 1/2 player ping,
+ * unless that verb is queued for the next tick in which case its some number probably smaller than world.tick_lag.
+ * so some verbs that represent player input are important enough that we only introduce semantic latency if we absolutely need to.
+ * its for this reason why player clicks are handled in SSinput before even movement - semantic latency could cause someone to move out of range
+ * when the verb finally processes but it was in range if the verb had processed immediately and overtimed.
+ */
+
+///queuing tick_usage threshold for verbs that are high enough priority that they only queue if the server is overtiming.
+///ONLY use for critical verbs
+#define VERB_OVERTIME_QUEUE_THRESHOLD 100
+///queuing tick_usage threshold for verbs that need lower latency more than most verbs.
+#define VERB_HIGH_PRIORITY_QUEUE_THRESHOLD 95
+///default queuing tick_usage threshold for most verbs which can allow a small amount of latency to be processed in the next tick
+#define VERB_DEFAULT_QUEUE_THRESHOLD 85
+
+///attempt to queue this verb process if the server is overloaded. evaluates to FALSE if queuing isnt necessary or if it failed.
+///_verification_args... are only necessary if the verb_manager subsystem youre using checks them in can_queue_verb()
+///if you put anything in _verification_args that ISNT explicitely put in the can_queue_verb() override of the subsystem youre using,
+///it will runtime.
+#define TRY_QUEUE_VERB(_verb_callback, _tick_check, _subsystem_to_use, _verification_args...) (_queue_verb(_verb_callback, _tick_check, _subsystem_to_use, _verification_args))
+///queue wrapper for TRY_QUEUE_VERB() when you want to call the proc if the server isnt overloaded enough to queue
+#define QUEUE_OR_CALL_VERB(_verb_callback, _tick_check, _subsystem_to_use, _verification_args...) \
+	if(!TRY_QUEUE_VERB(_verb_callback, _tick_check, _subsystem_to_use, _verification_args)) {\
+		_verb_callback:InvokeAsync() \
+		};
+
+//goes straight to SSverb_manager with default tick threshold
+#define DEFAULT_TRY_QUEUE_VERB(_verb_callback, _verification_args...) (TRY_QUEUE_VERB(_verb_callback, VERB_DEFAULT_QUEUE_THRESHOLD, null, _verification_args))
+#define DEFAULT_QUEUE_OR_CALL_VERB(_verb_callback, _verification_args...) QUEUE_OR_CALL_VERB(_verb_callback, VERB_DEFAULT_QUEUE_THRESHOLD, null, _verification_args)
+
+//default tick threshold but nondefault subsystem
+#define TRY_QUEUE_VERB_FOR(_verb_callback, _subsystem_to_use, _verification_args...) (TRY_QUEUE_VERB(_verb_callback, VERB_DEFAULT_QUEUE_THRESHOLD, _subsystem_to_use, _verification_args))
+#define QUEUE_OR_CALL_VERB_FOR(_verb_callback, _subsystem_to_use, _verification_args...) QUEUE_OR_CALL_VERB(_verb_callback, VERB_DEFAULT_QUEUE_THRESHOLD, _subsystem_to_use, _verification_args)

code/__HELPERS/turfs.dm

@@ -211,6 +211,11 @@ Turf and target are separate in case you want to teleport some distance from a t
 	if(!istype(checked_atom))
 		return
 
+	//Find coordinates
+	var/turf/atom_turf = get_turf(checked_atom) //use checked_atom's turfs, as it's coords are the same as checked_atom's AND checked_atom's coords are lost if it is inside another atom
+	if(!atom_turf)
+		return null
+
 	//Find checked_atom's matrix so we can use it's X/Y pixel shifts
 	var/matrix/atom_matrix = matrix(checked_atom.transform)
 
@@ -229,10 +234,6 @@ Turf and target are separate in case you want to teleport some distance from a t
 	var/rough_x = round(round(pixel_x_offset, world.icon_size) / world.icon_size)
 	var/rough_y = round(round(pixel_y_offset, world.icon_size) / world.icon_size)
 
-	//Find coordinates
-	var/turf/atom_turf = get_turf(checked_atom) //use checked_atom's turfs, as it's coords are the same as checked_atom's AND checked_atom's coords are lost if it is inside another atom
-	if(!atom_turf)
-		return null
 	var/final_x = clamp(atom_turf.x + rough_x, 1, world.maxx)
 	var/final_y = clamp(atom_turf.y + rough_y, 1, world.maxy)
 

code/_onclick/click.dm

@@ -37,9 +37,10 @@
  *
  * Note that this proc can be overridden, and is in the case of screen objects.
  */
-/atom/Click(location,control,params)
+/atom/Click(location, control, params)
 	if(flags_1 & INITIALIZED_1)
 		SEND_SIGNAL(src, COMSIG_CLICK, location, control, params, usr)
+
 		usr.ClickOn(src, params)
 
 /atom/DblClick(location,control,params)

code/controllers/master.dm

@@ -378,6 +378,7 @@ GLOBAL_REAL(Master, /datum/controller/master) = new
 	while (1)
 		tickdrift = max(0, MC_AVERAGE_FAST(tickdrift, (((REALTIMEOFDAY - init_timeofday) - (world.time - init_time)) / world.tick_lag)))
 		var/starting_tick_usage = TICK_USAGE
+
 		if (init_stage != init_stage_completed)
 			return MC_LOOP_RTN_NEWSTAGES
 		if (processing <= 0)

code/controllers/subsystem/input.dm

@@ -1,15 +1,28 @@
-SUBSYSTEM_DEF(input)
+VERB_MANAGER_SUBSYSTEM_DEF(input)
 	name = "Input"
-	wait = 1 //SS_TICKER means this runs every tick
 	init_order = INIT_ORDER_INPUT
 	init_stage = INITSTAGE_EARLY
 	flags = SS_TICKER
 	priority = FIRE_PRIORITY_INPUT
 	runlevels = RUNLEVELS_DEFAULT | RUNLEVEL_LOBBY
 
+	use_default_stats = FALSE
+
 	var/list/macro_set
 
-/datum/controller/subsystem/input/Initialize()
+	///running average of how many clicks inputted by a player the server processes every second. used for the subsystem stat entry
+	var/clicks_per_second = 0
+	///count of how many clicks onto atoms have elapsed before being cleared by fire(). used to average with clicks_per_second.
+	var/current_clicks = 0
+	///acts like clicks_per_second but only counts the clicks actually processed by SSinput itself while clicks_per_second counts all clicks
+	var/delayed_clicks_per_second = 0
+	///running average of how many movement iterations from player input the server processes every second. used for the subsystem stat entry
+	var/movements_per_second = 0
+	///running average of the amount of real time clicks take to truly execute after the command is originally sent to the server.
+	///if a click isnt delayed at all then it counts as 0 deciseconds.
+	var/average_click_delay = 0
+
+/datum/controller/subsystem/verb_manager/input/Initialize()
 	setup_default_macro_sets()
 
 	initialized = TRUE
@@ -19,7 +32,7 @@ SUBSYSTEM_DEF(input)
 	return ..()
 
 // This is for when macro sets are eventualy datumized
-/datum/controller/subsystem/input/proc/setup_default_macro_sets()
+/datum/controller/subsystem/verb_manager/input/proc/setup_default_macro_sets()
 	macro_set = list(
 	"Any" = "\"KeyDown \[\[*\]\]\"",
 	"Any+UP" = "\"KeyUp \[\[*\]\]\"",
@@ -29,12 +42,57 @@ SUBSYSTEM_DEF(input)
 	)
 
 // Badmins just wanna have fun ♪
-/datum/controller/subsystem/input/proc/refresh_client_macro_sets()
+/datum/controller/subsystem/verb_manager/input/proc/refresh_client_macro_sets()
 	var/list/clients = GLOB.clients
 	for(var/i in 1 to clients.len)
 		var/client/user = clients[i]
 		user.set_macros()
 
-/datum/controller/subsystem/input/fire()
-	for(var/mob/user as anything in GLOB.keyloop_list)
-		user.focus?.keyLoop(user.client)
+/datum/controller/subsystem/verb_manager/input/can_queue_verb(datum/callback/verb_callback/incoming_callback, control)
+	//make sure the incoming verb is actually something we specifically want to handle
+	if(control != "mapwindow.map")
+		return FALSE
+
+	if(average_click_delay >= MAXIMUM_CLICK_LATENCY || !..())
+		current_clicks++
+		average_click_delay = MC_AVG_FAST_UP_SLOW_DOWN(average_click_delay, 0)
+		return FALSE
+
+	return TRUE
+
+///stupid workaround for byond not recognizing the /atom/Click typepath for the queued click callbacks
+/atom/proc/_Click(location, control, params)
+	if(usr)
+		Click(location, control, params)
+
+
+/datum/controller/subsystem/verb_manager/input/fire()
+	var/moves_this_run = 0
+	var/deferred_clicks_this_run = 0 //acts like current_clicks but doesnt count clicks that dont get processed by SSinput
+
+	for(var/datum/callback/verb_callback/queued_click as anything in verb_queue)
+		if(!istype(queued_click))
+			stack_trace("non /datum/callback/verb_callback instance inside SSinput's verb_queue!")
+			continue
+
+		average_click_delay = MC_AVG_FAST_UP_SLOW_DOWN(average_click_delay, (REALTIMEOFDAY - queued_click.creation_time) SECONDS)
+		queued_click.InvokeAsync()
+
+		current_clicks++
+		deferred_clicks_this_run++
+
+	verb_queue.Cut() //is ran all the way through every run, no exceptions
+
+	for(var/mob/user in GLOB.keyloop_list)
+		moves_this_run += user.focus?.keyLoop(user.client)//only increments if a player changes their movement input from the last tick
+
+	clicks_per_second = MC_AVG_SECONDS(clicks_per_second, current_clicks, wait TICKS)
+	delayed_clicks_per_second = MC_AVG_SECONDS(delayed_clicks_per_second, deferred_clicks_this_run, wait TICKS)
+	movements_per_second = MC_AVG_SECONDS(movements_per_second, moves_this_run, wait TICKS)
+
+	current_clicks = 0
+
+/datum/controller/subsystem/verb_manager/input/stat_entry(msg)
+	. = ..()
+	. += "M/S:[round(movements_per_second,0.01)] | C/S:[round(clicks_per_second,0.01)]([round(delayed_clicks_per_second,0.01)] | CD: [round(average_click_delay,0.01)])"
+

code/controllers/subsystem/speech_controller.dm

@@ -1,53 +1,5 @@
-SUBSYSTEM_DEF(speech_controller)
+/// verb_manager subsystem just for handling say's
+VERB_MANAGER_SUBSYSTEM_DEF(speech_controller)
 	name = "Speech Controller"
 	wait = 1
-	flags = SS_TICKER|SS_NO_INIT
 	priority = FIRE_PRIORITY_SPEECH_CONTROLLER//has to be high priority, second in priority ONLY to SSinput
-	runlevels = RUNLEVELS_DEFAULT | RUNLEVEL_LOBBY
-
-	///used so that an admin can force all speech verbs to execute immediately instead of queueing
-	var/FOR_ADMINS_IF_BROKE_immediately_execute_all_speech = FALSE
-
-	///list of the form: list(client mob, message that mob is queued to say, other say arguments (if any)).
-	///this is our process queue, processed every tick.
-	var/list/queued_says_to_execute = list()
-
-///queues mob_to_queue into our process list so they say(message) near the start of the next tick
-/datum/controller/subsystem/speech_controller/proc/queue_say_for_mob(mob/mob_to_queue, message, message_type)
-
-	if(!TICK_CHECK || FOR_ADMINS_IF_BROKE_immediately_execute_all_speech)
-		process_single_say(mob_to_queue, message, message_type)
-		return TRUE
-
-	queued_says_to_execute += list(list(mob_to_queue, message, message_type))
-
-	return TRUE
-
-/datum/controller/subsystem/speech_controller/fire(resumed)
-
-	///	cache for sanic speed (lists are references anyways)
-	var/list/says_to_process = queued_says_to_execute.Copy()
-	queued_says_to_execute.Cut()//we should be going through the entire list every single iteration
-
-	for(var/list/say_to_process as anything in says_to_process)
-
-		var/mob/mob_to_speak = say_to_process[MOB_INDEX]//index 1 is the mob, 2 is the message, 3 is the message category
-		var/message = say_to_process[MESSAGE_INDEX]
-		var/message_category = say_to_process[CATEGORY_INDEX]
-
-		process_single_say(mob_to_speak, message, message_category)
-
-///used in fire() to process a single mobs message through the relevant proc.
-///only exists so that sleeps in the message pipeline dont cause the whole queue to wait
-/datum/controller/subsystem/speech_controller/proc/process_single_say(mob/mob_to_speak, message, message_category)
-	set waitfor = FALSE
-
-	switch(message_category)
-		if(SPEECH_CONTROLLER_QUEUE_SAY_VERB)
-			mob_to_speak.say(message)
-
-		if(SPEECH_CONTROLLER_QUEUE_WHISPER_VERB)
-			mob_to_speak.whisper(message)
-
-		if(SPEECH_CONTROLLER_QUEUE_EMOTE_VERB)
-			mob_to_speak.emote("me",1,message,TRUE)

code/controllers/subsystem/verb_manager.dm

@@ -0,0 +1,121 @@
+/**
+ * SSverb_manager, a subsystem that runs every tick and runs through its entire queue without yielding like SSinput.
+ * this exists because of how the byond tick works and where user inputted verbs are put within it.
+ *
+ * see TICK_ORDER.md for more info on how the byond tick is structured.
+ *
+ * The way the MC allots its time is via TICK_LIMIT_RUNNING, it simply subtracts the cost of SendMaps (MAPTICK_LAST_INTERNAL_TICK_USAGE)
+ * plus TICK_BYOND_RESERVE from the tick and uses up to that amount of time (minus the percentage of the tick used by the time it executes subsystems)
+ * on subsystems running cool things like atmospherics or Life or SSInput or whatever.
+ *
+ * Without this subsystem, verbs are likely to cause overtime if the MC uses all of the time it has alloted for itself in the tick, and SendMaps
+ * uses as much as its expected to, and an expensive verb ends up executing that tick. This is because the MC is completely blind to the cost of
+ * verbs, it can't account for it at all. The only chance for verbs to not cause overtime in a tick where the MC used as much of the tick
+ * as it alloted itself and where SendMaps costed as much as it was expected to is if the verb(s) take less than TICK_BYOND_RESERVE percent of
+ * the tick, which isnt much. Not to mention if SendMaps takes more than 30% of the tick and the MC forces itself to take at least 70% of the
+ * normal tick duration which causes ticks to naturally overrun even in the absence of verbs.
+ *
+ * With this subsystem, the MC can account for the cost of verbs and thus stop major overruns of ticks. This means that the most important subsystems
+ * like SSinput can start at the same time they were supposed to, leading to a smoother experience for the player since ticks arent riddled with
+ * minor hangs over and over again.
+ */
+SUBSYSTEM_DEF(verb_manager)
+	name = "Verb Manager"
+	wait = 1
+	flags = SS_TICKER | SS_NO_INIT
+	priority = FIRE_PRIORITY_DELAYED_VERBS
+	runlevels = RUNLEVEL_INIT | RUNLEVELS_DEFAULT
+
+	///list of callbacks to procs called from verbs or verblike procs that were executed when the server was overloaded and had to delay to the next tick.
+	///this list is ran through every tick, and the subsystem does not yield until this queue is finished.
+	var/list/datum/callback/verb_callback/verb_queue = list()
+
+	///running average of how many verb callbacks are executed every second. used for the stat entry
+	var/verbs_executed_per_second = 0
+
+	///if TRUE we treat usr's with holders just like usr's without holders. otherwise they always execute immediately
+	var/can_queue_admin_verbs = FALSE
+
+	///if this is true all verbs immediately execute and dont queue. in case the mc is fucked or something
+	var/FOR_ADMINS_IF_VERBS_FUCKED_immediately_execute_all_verbs = FALSE
+
+	///used for subtypes to determine if they use their own stats for the stat entry
+	var/use_default_stats = TRUE
+
+	///if TRUE this will... message admins every time a verb is queued to this subsystem for the next tick with stats.
+	///for obvious reasons dont make this be TRUE on the code level this is for admins to turn on
+	var/message_admins_on_queue = FALSE
+
+/**
+ * queue a callback for the given verb/verblike proc and any given arguments to the specified verb subsystem, so that they process in the next tick.
+ * intended to only work with verbs or verblike procs called directly from client input, use as part of TRY_QUEUE_VERB() and co.
+ *
+ * returns TRUE if the queuing was successful, FALSE otherwise.
+ */
+/proc/_queue_verb(datum/callback/verb_callback/incoming_callback, tick_check, datum/controller/subsystem/verb_manager/subsystem_to_use = SSverb_manager, ...)
+	if(TICK_USAGE < tick_check \
+	|| QDELETED(incoming_callback) \
+	|| QDELETED(incoming_callback.object) \
+	|| !ismob(usr) \
+	|| QDELING(usr))
+		return FALSE
+
+	if(!istype(subsystem_to_use))
+		return FALSE
+
+	var/list/args_to_check = args.Copy()
+	args_to_check.Cut(2, 4)//cut out tick_check and subsystem_to_use
+
+	//any subsystem can use the additional arguments to refuse queuing
+	if(!subsystem_to_use.can_queue_verb(arglist(args_to_check)))
+		return FALSE
+
+	return subsystem_to_use.queue_verb(incoming_callback)
+
+/**
+ * subsystem-specific check for whether a callback can be queued.
+ * intended so that subsystem subtypes can verify whether
+ *
+ * subtypes may include additional arguments here if they need them! you just need to include them properly
+ * in TRY_QUEUE_VERB() and co.
+ */
+/datum/controller/subsystem/verb_manager/proc/can_queue_verb(datum/callback/verb_callback/incoming_callback)
+	if(usr.client?.holder && !can_queue_admin_verbs \
+	|| FOR_ADMINS_IF_VERBS_FUCKED_immediately_execute_all_verbs \
+	|| !initialized \
+	|| !(runlevels & Master.current_runlevel))
+		return FALSE
+
+	return TRUE
+
+/**
+ * queue a callback for the given proc, so that it is invoked in the next tick.
+ * intended to only work with verbs or verblike procs called directly from client input, use as part of TRY_QUEUE_VERB()
+ *
+ * returns TRUE if the queuing was successful, FALSE otherwise.
+ */
+/datum/controller/subsystem/verb_manager/proc/queue_verb(datum/callback/verb_callback/incoming_callback)
+	. = FALSE //errored
+	if(message_admins_on_queue)
+		message_admins("[name] verb queuing: tick usage: [TICK_USAGE]%, proc: [incoming_callback.delegate], object: [incoming_callback.object], usr: [usr]")
+	verb_queue += incoming_callback
+	return TRUE
+
+/datum/controller/subsystem/verb_manager/fire(resumed)
+	var/executed_verbs = 0
+
+	for(var/datum/callback/verb_callback/verb_callback as anything in verb_queue)
+		if(!istype(verb_callback))
+			stack_trace("non /datum/callback/verb_callback inside [name]'s verb_queue!")
+			continue
+
+		verb_callback.InvokeAsync()
+		executed_verbs++
+
+	verb_queue.Cut()
+	verbs_executed_per_second = MC_AVG_SECONDS(verbs_executed_per_second, executed_verbs, wait TICKS)
+
+/datum/controller/subsystem/verb_manager/stat_entry(msg)
+	. = ..()
+	if(use_default_stats)
+		. += "V/S: [round(verbs_executed_per_second, 0.01)]"

code/datums/verb_callbacks.dm

@@ -0,0 +1,8 @@
+///like normal callbacks but they also record their creation time for measurement purposes
+/datum/callback/verb_callback
+	///the REALTIMEOFDAY this callback datum was created in. used for testing latency
+	var/creation_time = 0
+
+/datum/callback/verb_callback/New(thingtocall, proctocall, ...)
+	creation_time = REALTIMEOFDAY
+	. = ..()

code/modules/client/client_procs.dm

@@ -128,8 +128,17 @@ GLOBAL_LIST_INIT(blacklisted_builds, list(
 		if(QDELETED(real_src))
 			return
 
+	//fun fact: Topic() acts like a verb and is executed at the end of the tick like other verbs. So we have to queue it if the server is
+	//overloaded
+	if(hsrc && hsrc != holder && DEFAULT_TRY_QUEUE_VERB(VERB_CALLBACK(src, .proc/_Topic, hsrc, href, href_list)))
+		return
 	..() //redirect to hsrc.Topic()
 
+///dumb workaround because byond doesnt seem to recognize the .proc/Topic() typepath for /datum/proc/Topic() from the client Topic,
+///so we cant queue it without this
+/client/proc/_Topic(datum/hsrc, href, list/href_list)
+	return hsrc.Topic(href, href_list)
+
 /client/proc/is_content_unlocked()
 	if(!prefs.unlock_content)
 		to_chat(src, "Become a BYOND member to access member-perks and features, as well as support the engine that makes this game possible. Only 10 bucks for 3 months! <a href=\"https://secure.byond.com/membership\">Click Here to find out more</a>.")
@@ -915,6 +924,7 @@ GLOBAL_LIST_INIT(blacklisted_builds, list(
 			click_intercept_time = 0 //Reset and return. Next click should work, but not this one.
 			return
 		click_intercept_time = 0 //Just reset. Let's not keep re-checking forever.
+
 	var/ab = FALSE
 	var/list/modifiers = params2list(params)
 
@@ -928,12 +938,16 @@ GLOBAL_LIST_INIT(blacklisted_builds, list(
 	var/mcl = CONFIG_GET(number/minute_click_limit)
 	if (!holder && mcl)
 		var/minute = round(world.time, 600)
+
 		if (!clicklimiter)
 			clicklimiter = new(LIMITER_SIZE)
+
 		if (minute != clicklimiter[CURRENT_MINUTE])
 			clicklimiter[CURRENT_MINUTE] = minute
 			clicklimiter[MINUTE_COUNT] = 0
-		clicklimiter[MINUTE_COUNT] += 1+(ab)
+
+		clicklimiter[MINUTE_COUNT] += 1 + (ab)
+
 		if (clicklimiter[MINUTE_COUNT] > mcl)
 			var/msg = "Your previous click was ignored because you've done too many in a minute."
 			if (minute != clicklimiter[ADMINSWARNED_AT]) //only one admin message per-minute. (if they spam the admins can just boot/ban them)
@@ -954,14 +968,22 @@ GLOBAL_LIST_INIT(blacklisted_builds, list(
 		var/second = round(world.time, 10)
 		if (!clicklimiter)
 			clicklimiter = new(LIMITER_SIZE)
+
 		if (second != clicklimiter[CURRENT_SECOND])
 			clicklimiter[CURRENT_SECOND] = second
 			clicklimiter[SECOND_COUNT] = 0
-		clicklimiter[SECOND_COUNT] += 1+(!!ab)
+
+		clicklimiter[SECOND_COUNT] += 1 + (!!ab)
+
 		if (clicklimiter[SECOND_COUNT] > scl)
 			to_chat(src, span_danger("Your previous click was ignored because you've done too many in a second"))
 			return
 
+	//check if the server is overloaded and if it is then queue up the click for next tick
+	//yes having it call a wrapping proc on the subsystem is fucking stupid glad we agree unfortunately byond insists its reasonable
+	if(TRY_QUEUE_VERB(VERB_CALLBACK(object, /atom/proc/_Click, location, control, params), VERB_HIGH_PRIORITY_QUEUE_THRESHOLD, SSinput, control))
+		return
+
 	if (hotkeys)
 		// If hotkey mode is enabled, then clicking the map will automatically
 		// unfocus the text bar.

code/modules/keybindings/bindings_atom.dm

@@ -22,3 +22,6 @@
 		keybind_face_direction(movement_dir)
 	else
 		user?.Move(get_step(src, movement_dir), movement_dir)
+		return !!movement_dir //true if there was actually any player input
+
+	return FALSE

code/modules/mob/dead/observer/observer.dm

@@ -696,14 +696,11 @@ This is the proc mobs get to turn into a ghost. Forked from ghostize due to comp
 	target.faction = list("neutral")
 	return TRUE
 
-//this is a mob verb instead of atom for performance reasons
-//see /mob/verb/examinate() in mob.dm for more info
-//overridden here and in /mob/living for different point span classes and sanity checks
-/mob/dead/observer/pointed(atom/A as mob|obj|turf in view(client.view, src))
+/mob/dead/observer/_pointed(atom/pointed_at)
 	if(!..())
 		return FALSE
-	usr.visible_message(span_deadsay("<b>[src]</b> points to [A]."))
-	return TRUE
+
+	usr.visible_message(span_deadsay("<b>[src]</b> points to [pointed_at]."))
 
 /mob/dead/observer/verb/view_manifest()
 	set name = "View Crew Manifest"

code/modules/mob/inventory.dm

@@ -428,6 +428,10 @@
 	set name = "quick-equip"
 	set hidden = TRUE
 
+	DEFAULT_QUEUE_OR_CALL_VERB(VERB_CALLBACK(src, .proc/execute_quick_equip))
+
+///proc extender of [/mob/verb/quick_equip] used to make the verb queuable if the server is overloaded
+/mob/proc/execute_quick_equip()
 	var/obj/item/I = get_active_held_item()
 	if(!I)
 		to_chat(src, span_warning("You are not holding anything to equip!"))

code/modules/mob/living/living.dm

@@ -449,12 +449,14 @@
 /mob/living/pointed(atom/A as mob|obj|turf in view(client.view, src))
 	if(incapacitated())
 		return FALSE
+
+	return ..()
+
+/mob/living/_pointed(atom/pointing_at)
 	if(!..())
 		return FALSE
-	visible_message("<span class='infoplain'>[span_name("[src]")] points at [A].</span>", span_notice("You point at [A]."))
-	log_message("points at [A]", LOG_EMOTE)
-	return TRUE
-
+	log_message("points at [pointing_at]", LOG_EMOTE)
+	visible_message("<span class='infoplain'>[span_name("[src]")] points at [pointing_at].</span>", span_notice("You point at [pointing_at]."))
 
 /mob/living/verb/succumb(whispered as null)
 	set hidden = TRUE
@@ -1038,6 +1040,10 @@
 	set name = "Resist"
 	set category = "IC"
 
+	DEFAULT_QUEUE_OR_CALL_VERB(VERB_CALLBACK(src, .proc/execute_resist))
+
+///proc extender of [/mob/living/verb/resist] meant to make the process queable if the server is overloaded when the verb is called
+/mob/living/proc/execute_resist()
 	if(!can_resist())
 		return
 	changeNext_move(CLICK_CD_RESIST)
@@ -1063,7 +1069,6 @@
 		else if(last_special <= world.time)
 			resist_restraints() //trying to remove cuffs.
 
-
 /mob/proc/resist_grab(moving_resist)
 	return 1 //returning 0 means we successfully broke free
 

code/modules/mob/living/silicon/robot/robot.dm

@@ -421,6 +421,9 @@
 	set category = "IC"
 	set src = usr
 
+	return ..()
+
+/mob/living/silicon/robot/execute_mode()
 	if(incapacitated())
 		return
 	var/obj/item/W = get_active_held_item()
@@ -947,12 +950,11 @@
 	buckle_mob_flags= RIDER_NEEDS_ARM // just in case
 	return ..()
 
-/mob/living/silicon/robot/resist()
+/mob/living/silicon/robot/execute_resist()
 	. = ..()
 	if(!has_buckled_mobs())
 		return
-	for(var/i in buckled_mobs)
-		var/mob/unbuckle_me_now = i
+	for(var/mob/unbuckle_me_now as anything in buckled_mobs)
 		unbuckle_mob(unbuckle_me_now, FALSE)
 
 

code/modules/mob/living/simple_animal/bot/mulebot.dm

@@ -753,8 +753,8 @@
 /mob/living/simple_animal/bot/mulebot/remove_air(amount) //To prevent riders suffocating
 	return loc ? loc.remove_air(amount) : null
 
-/mob/living/simple_animal/bot/mulebot/resist()
-	..()
+/mob/living/simple_animal/bot/mulebot/execute_resist()
+	. = ..()
 	if(load)
 		unload()
 

code/modules/mob/mob.dm

@@ -526,6 +526,10 @@
 	set name = "Examine"
 	set category = "IC"
 
+	DEFAULT_QUEUE_OR_CALL_VERB(VERB_CALLBACK(src, .proc/run_examinate, examinify))
+
+/mob/proc/run_examinate(atom/examinify)
+
 	if(isturf(examinify) && !(sight & SEE_TURFS) && !(examinify in view(client ? client.view : world.view, src)))
 		// shift-click catcher may issue examinate() calls for out-of-sight turfs
 		return
@@ -721,6 +725,10 @@
 	set category = "Object"
 	set src = usr
 
+	DEFAULT_QUEUE_OR_CALL_VERB(VERB_CALLBACK(src, .proc/execute_mode))
+
+///proc version to finish /mob/verb/mode() execution. used in case the proc needs to be queued for the tick after its first called
+/mob/proc/execute_mode()
 	if(ismecha(loc))
 		return
 
@@ -1125,7 +1133,8 @@
 /mob/proc/update_mouse_pointer()
 	if(!client)
 		return
-	client.mouse_pointer_icon = initial(client.mouse_pointer_icon)
+	if(client.mouse_pointer_icon != initial(client.mouse_pointer_icon))//only send changes to the client if theyre needed
+		client.mouse_pointer_icon = initial(client.mouse_pointer_icon)
 	if(examine_cursor_icon && client.keys_held["Shift"]) //mouse shit is hardcoded, make this non hard-coded once we make mouse modifiers bindable
 		client.mouse_pointer_icon = examine_cursor_icon
 	if(istype(loc, /obj/vehicle/sealed))

code/modules/mob/mob_say.dm

@@ -13,7 +13,7 @@
 	//queue this message because verbs are scheduled to process after SendMaps in the tick and speech is pretty expensive when it happens.
 	//by queuing this for next tick the mc can compensate for its cost instead of having speech delay the start of the next tick
 	if(message)
-		SSspeech_controller.queue_say_for_mob(src, message, SPEECH_CONTROLLER_QUEUE_SAY_VERB)
+		QUEUE_OR_CALL_VERB_FOR(VERB_CALLBACK(src, /atom/movable/proc/say, message), SSspeech_controller)
 
 ///Whisper verb
 /mob/verb/whisper_verb(message as text)
@@ -26,7 +26,7 @@
 		return
 
 	if(message)
-		SSspeech_controller.queue_say_for_mob(src, message, SPEECH_CONTROLLER_QUEUE_WHISPER_VERB)
+		QUEUE_OR_CALL_VERB_FOR(VERB_CALLBACK(src, /mob/proc/whisper, message), SSspeech_controller)
 
 /**
  * Whisper a message.
@@ -49,7 +49,7 @@
 
 	message = trim(copytext_char(sanitize(message), 1, MAX_MESSAGE_LEN))
 
-	SSspeech_controller.queue_say_for_mob(src, message, SPEECH_CONTROLLER_QUEUE_EMOTE_VERB)
+	QUEUE_OR_CALL_VERB_FOR(VERB_CALLBACK(src, /mob/proc/emote, "me", 1, message, TRUE), SSspeech_controller)
 
 ///Speak as a dead person (ghost etc)
 /mob/proc/say_dead(message)

code/modules/point/point.dm

@@ -95,16 +95,22 @@
  *
  * overridden here and in /mob/dead/observer for different point span classes and sanity checks
  */
-/mob/verb/pointed(atom/target as mob|obj|turf in view())
+/mob/verb/pointed(atom/A as mob|obj|turf in view())
 	set name = "Point To"
 	set category = "Object"
 
-	if(client && !(target in view(client.view, src)))
-		return FALSE
-	if(istype(target, /obj/effect/temp_visual/point))
+	if(istype(A, /obj/effect/temp_visual/point))
 		return FALSE
 
-	point_at(target)
+	DEFAULT_QUEUE_OR_CALL_VERB(VERB_CALLBACK(src, .proc/_pointed, A))
 
-	SEND_SIGNAL(src, COMSIG_MOB_POINTED, target)
+/// possibly delayed verb that finishes the pointing process starting in [/mob/verb/pointed()].
+/// either called immediately or in the tick after pointed() was called, as per the [DEFAULT_QUEUE_OR_CALL_VERB()] macro
+/mob/proc/_pointed(atom/pointing_at)
+	if(client && !(pointing_at in view(client.view, src)))
+		return FALSE
+
+	point_at(pointing_at)
+
+	SEND_SIGNAL(src, COMSIG_MOB_POINTED, pointing_at)
 	return TRUE

code/modules/tgui/tgui.dm

@@ -325,8 +325,7 @@
 			window = window,
 			src_object = src_object)
 		process_status()
-		if(src_object.ui_act(act_type, payload, src, state))
-			SStgui.update_uis(src_object)
+		DEFAULT_QUEUE_OR_CALL_VERB(VERB_CALLBACK(src, .proc/on_act_message, act_type, payload, state))
 		return FALSE
 	switch(type)
 		if("ready")
@@ -349,3 +348,10 @@
 			LAZYINITLIST(src_object.tgui_shared_states)
 			src_object.tgui_shared_states[href_list["key"]] = href_list["value"]
 			SStgui.update_uis(src_object)
+
+/// Wrapper for behavior to potentially wait until the next tick if the server is overloaded
+/datum/tgui/proc/on_act_message(act_type, payload, state)
+	if(QDELETED(src) || QDELETED(src_object))
+		return
+	if(src_object.ui_act(act_type, payload, src, state))
+		SStgui.update_uis(src_object)

tgstation.dme

@@ -92,6 +92,7 @@
 #include "code\__DEFINES\important_recursive_contents.dm"
 #include "code\__DEFINES\industrial_lift.dm"
 #include "code\__DEFINES\injection.dm"
+#include "code\__DEFINES\input.dm"
 #include "code\__DEFINES\instruments.dm"
 #include "code\__DEFINES\interaction_flags.dm"
 #include "code\__DEFINES\inventory.dm"
@@ -190,6 +191,7 @@
 #include "code\__DEFINES\typeids.dm"
 #include "code\__DEFINES\uplink.dm"
 #include "code\__DEFINES\vehicles.dm"
+#include "code\__DEFINES\verb_manager.dm"
 #include "code\__DEFINES\vv.dm"
 #include "code\__DEFINES\wall_dents.dm"
 #include "code\__DEFINES\weather.dm"
@@ -605,6 +607,7 @@
 #include "code\controllers\subsystem\timer.dm"
 #include "code\controllers\subsystem\title.dm"
 #include "code\controllers\subsystem\traitor.dm"
+#include "code\controllers\subsystem\verb_manager.dm"
 #include "code\controllers\subsystem\vis_overlays.dm"
 #include "code\controllers\subsystem\vote.dm"
 #include "code\controllers\subsystem\wardrobe.dm"
@@ -676,6 +679,7 @@
 #include "code\datums\station_alert.dm"
 #include "code\datums\station_integrity.dm"
 #include "code\datums\tgs_event_handler.dm"
+#include "code\datums\verb_callbacks.dm"
 #include "code\datums\verbs.dm"
 #include "code\datums\view.dm"
 #include "code\datums\voice_of_god_command.dm"