Unicode awareness Part 2 -- copytext() (#48512)

* Unicode support Part 2 -- copytext()

This is the transition of all copytext() calls to be unicode aware and also some nearby calls in the same functions. Most things are just replacing copytext() with copytext_char() as a terrible character limiter but a few others were slightly more involved.

I replaced a ton of
````
var/something = sanitize(input())
something = copytext(something, 1, MAX_MESSAGE_LEN)
````

with a single stripped_input() call. stripped_input() already calls html_encode(), trim(), and some other sanitization so there shouldn't be any major issues there.

This is still VERY rough btw; DNA is a mess, the status displays are complete ass, there's a copytext() in code\datums\shuttles.dm that I'm not sure what to do with, and I didn't touch anything in the tools folder. I haven't tested this much at all yet, I only got it to compile earlier this morning. There's also likely to be weird bugs until I get around to fixing length(), findtext(), and the rest of the string procs.

* Makes the code functional

* Assume color hex strings are always # followed by ascii.
Properly encodes and decodes the stuff in mob_helpers.dm which fixes some issues there.

* Removes ninjaspeak since it's unused
This commit is contained in:
MrPerson
2020-01-18 13:07:22 +13:00
committed by oranges
parent e7a9bf4a6d
commit f7eb2c905b
110 changed files with 501 additions and 557 deletions
@@ -260,7 +260,7 @@
node += "*"
i++
else if (copytext(token(i), 1, 2) == "/")
else if(token(i)[1] == "/")
i = object_type(i, node)
else
@@ -391,7 +391,7 @@
//object_type: <type path>
/datum/SDQL_parser/proc/object_type(i, list/node)
if (copytext(token(i), 1, 2) != "/")
if(token(i)[1] != "/")
return parse_error("Expected type, but it didn't begin with /")
var/path = text2path(token(i))
@@ -430,7 +430,7 @@
//string: ''' <some text> ''' | '"' <some text > '"'
/datum/SDQL_parser/proc/string(i, list/node)
if(copytext(token(i), 1, 2) in list("'", "\""))
if(token(i)[1] in list("'", "\""))
node += token(i)
else
@@ -441,7 +441,7 @@
//array: '[' expression_list ']'
/datum/SDQL_parser/proc/array(i, list/node)
// Arrays get turned into this: list("[", list(exp_1a = exp_1b, ...), ...), "[" is to mark the next node as an array.
if(copytext(token(i), 1, 2) != "\[")
if(token(i)[1] != "\[")
parse_error("Expected an array but found '[token(i)]'")
return i + 1
@@ -613,7 +613,7 @@
node += "null"
i++
else if(lowertext(copytext(token(i), 1, 3)) == "0x" && isnum(hex2num(copytext(token(i), 3))))
else if(lowertext(copytext(token(i), 1, 3)) == "0x" && isnum(hex2num(copytext(token(i), 3))))//3 == length("0x") + 1
node += hex2num(copytext(token(i), 3))
i++
@@ -621,16 +621,16 @@
node += text2num(token(i))
i++
else if(copytext(token(i), 1, 2) in list("'", "\""))
else if(token(i)[1] in list("'", "\""))
i = string(i, node)
else if(copytext(token(i), 1, 2) == "\[") // Start a list.
else if(token(i)[1] == "\[") // Start a list.
i = array(i, node)
else if(copytext(token(i), 1, 3) == "@\[")
else if(copytext(token(i), 1, 3) == "@\[")//3 == length("@\[") + 1
i = selectors_array(i, node)
else if(copytext(token(i), 1, 2) == "/")
else if(token(i)[1] == "/")
i = object_type(i, node)
else