Better SQL prepared statements (#2474)

The system used to be of complexity O(n^2). Essentially two for loops running per every argument. Which ended up being surprisingly slow (there were instances where I saw the argument parser as using quite a lot of CPU time).

This replaces it with a more linear algorithm. It's somewhere near O(n) where n is the length of the unparsed query. Which is more stable and faaaster. This comes with two changes, however:

Parameters inside the query now have to be delimited from both sides with : (colons). The alternative to this would be to use something like $n or just assume that space marks the end of a marker. Only the former is workable, the latter would break a few queries already.
Arguments in the argument array no longer have to be prefixed by : (colons). So, while in the query you would write :thing:, you'd initialize the array of args as: list("thing" = somevar). It could be made to work without it, but eh, I think this is fine.
Argument validation is slightly weaker. What I mean by this is that with the old system, unused keys would result in an error. This is no longer a thing. Missing keys will still result in an error, however.
One more improvement: double delimiting removes an edge case where if key A partially covers key B, depending on the order, key A would mangle key B.
Updated and tested all queries that I could find. So this should be good.
This commit is contained in:
skull132
2017-05-29 21:17:41 +03:00
committed by GitHub
parent b88d6aaadb
commit a3ec0cf45d
36 changed files with 366 additions and 333 deletions
+72 -33
View File
@@ -183,47 +183,86 @@ DBQuery/proc/SetConversion(column,conversion)
conversions.len = column
conversions[column] = conversion
/* Works similarly to the PDO object's Execute() method in PHP.
* Insert a list of keys/values, it searches the SQL syntax for the keys,
* and replaces them with sanitized versions of the values.
* Can be called independently, or through dbcon.Execute(), where the list would be the first argument.
* passNotFound controls whether or not is passes keys not found in the SQL query.
* Keys are /case-sensitive/, be careful!
* Returns the parsed SQL query upon completion.
* - Skull132
/**
* Automatic query parsing. Works similarly to any SQL prepared statement system.
*
* You pass in here a query which has special argument markers (we use :marker: style),
* and a map of argument -> content. Note that in this map, the argument key should not
* contain the colon delimiters. So while it's :marker: in the query, in the args list,
* it would be list("marker" = somevar).
*
* The parser will then crawl the input query and replace any placeholders it finds
* with automatically sanitized values. Values can even be lists, at which point a
* MySQL list is generated: (VALUE1, VALUE2).
*
* @param query_to_parse The query we will be parsing.
* @param argument_list A map of markers associated with their values. Values can
* be numeric, strings, null, or lists of primitives. In case of null, MySQL NULL is
* used. In case of a list of primitives, a MySQL comma delimited list is used instead.
*
* @return The parsed query upon success. null upon failure.
*/
/DBQuery/proc/parseArguments(var/query_to_parse = null, var/list/argument_list, var/pass_not_found = 0)
/DBQuery/proc/parseArguments(var/query_to_parse = null, var/list/argument_list)
if (!query_to_parse || !argument_list || !argument_list.len)
log_debug("parseArguments() failed! Improper arguments sent!")
return 0
log_debug("SQL ARGPARSE: Invalid arguments sent.")
return null
for (var/placeholder in argument_list)
if (!findtextEx(query_to_parse, placeholder))
if (pass_not_found)
var/parsed = ""
var/list/cache = list()
var/pos = 1
var/search = 0
var/curr_arg = ""
// We crawl the key list first. This is required to properly notice missing
// arguments/keys. Otherwise, list[non-key] will always result in null and
// will thus be populated as NULL. Which is bad.
for (var/key in argument_list)
var/argument = argument_list[key]
if (istext(argument))
cache[key] = "[dbcon.Quote(argument)]"
else if (isnum(argument))
cache[key] = "[argument]"
else if (istype(argument, /list))
cache[key] = parse_db_lists(argument)
else if (isnull(argument))
cache[key] = "NULL"
else
log_debug("SQL ARGPARSE: Cannot identify argument! [key]. Argument: [argument]")
return null
while (1)
search = findtext(query_to_parse, ":", pos)
parsed += copytext(query_to_parse, pos, search)
if (search)
pos = search
search = findtext(query_to_parse, ":", pos + 1)
if (search)
curr_arg = copytext(query_to_parse, pos + 1, search)
if (cache[curr_arg])
parsed += cache[curr_arg]
else
log_debug("SQL ARGPARSE: Unpopulated argument found in an SQL query.")
log_debug("SQL ARGPARSE: [curr_arg]. Query: [query_to_parse]")
return null
pos = search + 1
curr_arg = ""
continue
else
log_debug("parseArguments() failed! Key not found: [placeholder].")
return 0
parsed += copytext(query_to_parse, pos, search)
var/argument = argument_list[placeholder]
break
if (istext(argument))
argument = dbcon.Quote(argument)
else if (isnum(argument))
argument = "[argument]"
else if (istype(argument, /list))
argument = parse_db_lists(argument)
else if (isnull(argument))
argument = "NULL"
else
log_debug("parseArguments() failed! Cannot identify argument!")
log_debug("Placeholder: '[placeholder]'. Argument: '[argument]'")
return 0
query_to_parse = replacetextEx(query_to_parse, placeholder, argument)
return query_to_parse
return parsed
/**
* Generates a MySQL list from a list of primitives. Also escapes values.
*
* @param argument The list of primitives we want to generate a MySQL list from.
*
* @return A string representing the MySQL list if the parsing is successful.
* "NULL", the MySQL null value, if parsing fails for some reason.
*/
/DBQuery/proc/parse_db_lists(var/list/argument)
if (!argument || !istype(argument) || !argument.len)
return "NULL"