Getting rid of mm1 and moving merge conflict resolution to mm2 (#5617)

This PRs gets rid of mm1 and moves merge conflict resolution entirely to mm2.
This commit is contained in:
Mykhailo Bykhovtsev
2018-11-17 00:52:49 +02:00
committed by Erki
parent 18b387c32e
commit 43cd737a78
15 changed files with 59 additions and 397 deletions
-3
View File
@@ -1,3 +0,0 @@
@echo off
set MAPROOT="../../maps/"
python dmm2tgm.py %1 %MAPROOT%
-12
View File
@@ -1,12 +0,0 @@
@echo off
cd ../../maps/aurora
for /R %%f in (*.dmm) do copy "%%f" "%%f.backup"
cls
echo All dmm files in the maps/aurora directory have been backed up.
echo Now you can make your changes...
echo ---
echo Remember to run Run Map Merge - TGM.bat just before you commit your changes!
echo ---
pause
-12
View File
@@ -1,12 +0,0 @@
@echo off
cd ../../maps/exodus
for /R %%f in (*.dmm) do copy "%%f" "%%f.backup"
cls
echo All dmm files in the maps/exodus directory have been backed up.
echo Now you can make your changes...
echo ---
echo Remember to run Run Map Merge - TGM.bat just before you commit your changes!
echo ---
pause
@@ -1,12 +0,0 @@
@echo off
cd ../../maps/dungeon_spawns
for /R %%f in (*.dmm) do copy "%%f" "%%f.backup"
cls
echo All dmm files in the maps/dungeon_spawns directory have been backed up.
echo Now you can make your changes...
echo ---
echo Remember to run Run Map Merge - TGM.bat just before you commit your changes!
echo ---
pause
-12
View File
@@ -1,12 +0,0 @@
@echo off
cd ../../maps/runtime
for /R %%f in (*.dmm) do copy "%%f" "%%f.backup"
cls
echo All dmm files in the maps/runtime directory have been backed up.
echo Now you can make your changes...
echo ---
echo Remember to run Run Map Merge - TGM.bat just before you commit your changes!
echo ---
pause
-63
View File
@@ -1,63 +0,0 @@
# Map Merger
Before any change to a map, it is good to use the Map Merger tools. In a nutshell, it rewrites the map to minimize differences between different versions of the map (DreamMakers map editor rewrites a lot of the tile keys). This makes the git diff between different map changes much smaller. More recently a new way of laying out the map was invented by Remie, called TGM, this helps to further reduce conflicts in the map files.
This is good for a few reasons
- Maintainers can actually verify the changes you are making are what you say they are by simply viewing the diff (For small changes at least)
- The less changes there are in any given map diff, the easier it is for git to merge it without running into unexpected conflicts, which in most cases you have to either manually resolve or require you to remap your changes
However - to do all this is going to require you to put some elbow grease into understanding the map merger tool.
If you have difficulty using these tools, ask for help in #coderbus
## Using the tools
1. **Install Python 3.5 or greater** - If you don't have Python already installed it can be downloaded from: https://www.python.org/downloads/ - make sure you grab the latest python 3, again, it must be 3.5 or greater
2. **PATH Python** - This step is mostly applicable to windows users, you must make sure you ask the windows installer to add python to your path. If you have already installed python you may need to manually add it to your path as indicated in this guide
3. **Prepare Maps** - Run "Prepare Maps.bat" in the tools/mapmerge/ directory.
4. **Edit your map** - Make your changes to the map here. Remember to save them!
5. **Clean map** - Run "Run Map Merge - TGM.bat" in the tools/mapmerge/ directory.
6. **Check differences** - Use your git application of choice to look at the differences between revisions of your code and commit the result.
7. **Commit** - Your map is now ready to be committed, rejoice and wait for conflicts.
## Common pitfalls
Do *not* open the map in dreameditor before committing the results of the mapmerger - this can cause dreameditor to resave the map back to dmm, if you're having issues with your map getting stuck in dmm mode, try committing and pushing the mapmerger changes before reopening in dreameditor.
## Map Conflict Fixer/Helper
The map conflict fixer is a script that can help you fix map conflicts easier and faster. Here's how it works:
### Before using
You need git for this, of course. Make sure your development branch is up to date before starting a map edit to ensure the script outputs a correct fix.
### Dictionary mode
Dictionary conflicts are the easiest to fix, you simply need to create more models to accommodate your changes and everyone elses.
When you run in this mode, if the script finishes successfully the map should be ready to be committed.
If the script fails in dictionary mode, you can run it again in full fix mode.
### Full Fix mode
When you and someone else edit the same coordinate, there is no easy way to fix the conflict. You need to get your hands dirty.
The script will mark every tile with a marker type to help you identify what needs fixing in the map editor.
After you edit and fix a marked map, you should run it through the map merger. The .backup file should be the same you used before.
#### Priorities
In Full Fix mode, the script needs to know which map version has higher priority, yours or someone elses. This important so tiles with multiple area and turf types aren't created.
Your version has priority - In each conflicted coordinate, your floor type and your area type will be used Their version has priority - In each conflicted coordinate, your floor type and your area type will not be used
### IMPORTANT
This script is in a testing phase and you should not consider any output to be safe. Always verify the maps this script produced to make sure nothing is out of place.
-5
View File
@@ -1,5 +0,0 @@
@echo off
set MAPROOT="../../maps"
set TGM=0
python mapmerger.py %1 %MAPROOT% %TGM%
pause
-5
View File
@@ -1,5 +0,0 @@
@echo off
set MAPROOT="../../maps"
set TGM=1
python mapmerger.py %1 %MAPROOT% %TGM%
pause
-39
View File
@@ -1,39 +0,0 @@
import map_helpers
import sys
import shutil
#main("../../_maps/")
def main(map_folder):
tgm = "1"
maps = map_helpers.prompt_maps(map_folder, "convert", tgm)
print("\nConverting these maps:")
for i in maps.indices:
print(str(maps.files[i])[len(map_folder):])
convert = input("\nPress Enter to convert...\n")
if convert == "abort":
print("\nAborted map convert.")
sys.exit()
else:
for i in maps.indices:
path_str = str(maps.files[i])
path_str_pretty = path_str[len(map_folder):]
error = map_helpers.merge_map(path_str, path_str, tgm)
if error > 1:
print(map_helpers.error[error])
continue
if error == 1:
print(map_helpers.error[1])
print("CONVERTED: {}".format(path_str_pretty))
print(" - ")
print("\nFinished converting.")
def string_to_num(s):
try:
return int(s)
except ValueError:
return -1
main(sys.argv[1])
-46
View File
@@ -1,46 +0,0 @@
import map_helpers
import sys
import shutil
#main("../../_maps/")
def main(map_folder, tgm=0):
maps = map_helpers.prompt_maps(map_folder, "merge", tgm)
print("\nMerging these maps:")
for i in maps.indices:
print(str(maps.files[i])[len(map_folder):])
merge = input("\nPress Enter to merge...\n")
if merge == "abort":
print("\nAborted map merge.")
sys.exit()
else:
for i in maps.indices:
path_str = str(maps.files[i])
shutil.copyfile(path_str, path_str + ".before")
path_str_pretty = path_str[len(map_folder):]
try:
error = map_helpers.merge_map(path_str, path_str + ".backup", tgm)
if error > 1:
print(map_helpers.error[error])
os.remove(path_str + ".before")
continue
if error == 1:
print(map_helpers.error[1])
print("MERGED: {}".format(path_str_pretty))
print(" - ")
except FileNotFoundError:
print("ERROR: File not found! Make sure you run 'Prepare Maps.bat' before merging.")
print("MISSING BACKUP FILE: " + path_str_pretty + ".backup")
print(" - ")
print("\nFinished merging.")
print("\nNOTICE: A version of the map files from before merging have been created for debug purposes.\nDo not delete these files until it is sure your map edits have no undesirable changes.")
def string_to_num(s):
try:
return int(s)
except ValueError:
return -1
main(sys.argv[1], sys.argv[2])
-183
View File
@@ -1,183 +0,0 @@
import re
import os
import argparse
import map_helpers #why we don't have some generic package for these reee
default_map_directory = "../../_maps"
replacement_re = re.compile('\s*([^{]*)\s*(\{(.*)\})?')
def tgm_check(map_file):
with open(map_file) as f:
firstline = f.readline()
#why some maps have trailing spaces in this ???
if firstline.startswith(map_helpers.tgm_header):
return True
return False
def save_map(map_data,filepath,tgm=False):
map_data['dictionary'] = map_helpers.sort_dictionary(map_data['dictionary'])
if tgm:
map_helpers.write_dictionary_tgm(filepath, map_data['dictionary'],None)
map_helpers.write_grid_coord_small(filepath, map_data['grid'], map_data['maxx'], map_data['maxy'])
else:
map_helpers.write_dictionary(filepath, map_data['dictionary'],None)
map_helpers.write_grid(filepath, map_data['grid'], map_data['maxx'], map_data['maxy'])
def update_all_maps(update_file=None, update_string=None, map_directory=None, verbose=False):
if map_directory is None:
map_directory = os.path.normpath(os.path.join(os.path.dirname(__file__), default_map_directory))
for root, _, files in os.walk(map_directory):
for filepath in files:
if filepath.endswith(".dmm"):
path = os.path.join(root, filepath)
update_map(path, update_file, update_string, verbose)
def update_map(map_filepath, update_file=None, update_string=None, verbose=False):
print("Updating: {0}".format(map_filepath))
map_data = map_helpers.parse_map(map_helpers.get_map_raw_text(map_filepath))
tgm = tgm_check(map_filepath)
if update_file:
with open(update_file) as update_source:
for line in update_source:
if line.startswith("#") or line.isspace():
continue
map_data = update_path(map_data, line, verbose)
save_map(map_data, map_filepath, tgm)
if update_string:
map_data = update_path(map_data, update_string, verbose)
save_map(map_data, map_filepath, tgm)
def props_to_string(props):
return "{{{0}}}".format(";".join([k+" = "+props[k] for k in props]))
#urgent todo: replace with actual parser, this is slow as janitor in crit
split_re = re.compile('((?:[A-Za-z0-9_\-$]+)\s*=\s*(?:"(?:.+?)"|[^";]*)|@OLD)')
def string_to_props(propstring,verbose = False):
props = dict()
for raw_prop in re.split(split_re,propstring):
if not raw_prop or raw_prop.strip() == ';':
continue
prop = raw_prop.split('=', maxsplit=1)
props[prop[0].strip()] = prop[1].strip() if len(prop) > 1 else None
if verbose:
print("{0} to {1}".format(propstring,props))
return props
def parse_rep_string(replacement_string,verbose = False):
# translates /blah/blah {meme = "test",} into path,prop dictionary tuple
match = re.match(replacement_re, replacement_string)
path = match.group(1)
props = match.group(3)
if props:
prop_dict = string_to_props(props, verbose)
else:
prop_dict = dict()
return path.strip(), prop_dict
def update_path(mapdata, replacement_string, verbose=False):
old_path_part, new_path_part = replacement_string.split(':', maxsplit=1)
old_path, old_path_props = parse_rep_string(old_path_part,verbose)
new_paths = dict()
for replacement_def in new_path_part.split(','):
new_path, new_path_props = parse_rep_string(replacement_def,verbose)
new_paths[new_path] = new_path_props
def replace_def(match):
if match.group(2):
old_props = string_to_props(match.group(2),verbose)
else:
old_props = dict()
for filter_prop in old_path_props:
if filter_prop not in old_props:
if old_path_props[filter_prop] == "@UNSET":
continue
else:
return [match.group(0)]
else:
if old_props[filter_prop] != old_path_props[filter_prop] or old_path_props[filter_prop] == "@UNSET":
return [match.group(0)] #does not match current filter, skip the change.
if verbose:
print("Found match : {0}".format(match.group(0)))
out_paths = []
for new_path, new_props in new_paths.items():
out = new_path
out_props = dict()
for prop_name, prop_value in new_props.items():
if prop_name == "@OLD":
out_props = dict(old_props)
continue
if prop_value == "@SKIP":
out_props.pop(prop_name, None)
continue
if prop_value.startswith("@OLD"):
params = prop_value.split(":")
if prop_name in old_props:
out_props[prop_name] = old_props[params[1]] if len(params) > 1 else old_props[prop_name]
continue
out_props[prop_name] = prop_value
if out_props:
out += props_to_string(out_props)
out_paths.append(out)
if verbose:
print("Replacing with: {0}".format(out_paths))
return out_paths
def get_result(element):
p = re.compile("{0}\s*({{(.*)}})?$".format(re.escape(old_path)))
match = p.match(element)
if match:
return replace_def(match) # = re.sub(p,replace_def,element)
else:
return [element]
for definition_key in mapdata['dictionary']:
def_value = mapdata['dictionary'][definition_key]
start = list(def_value)
changed = [y for x in start for y in get_result(x)]
new_value = tuple(changed)
if new_value != def_value:
mapdata['dictionary'][definition_key] = new_value
return mapdata
if __name__ == "__main__":
desc = """
Update dmm files given update file/string.
Replacement syntax example:
/turf/open/floor/plasteel/warningline : /obj/effect/turf_decal {dir = @OLD ;tag = @SKIP;icon_state = @SKIP}
/turf/open/floor/plasteel/warningline : /obj/effect/turf_decal {@OLD} , /obj/thing {icon_state = @OLD:name; name = "meme"}
/turf/open/floor/plasteel/warningline{dir=2} : /obj/thing
New paths properties:
@OLD - if used as property name copies all modified properties from original path to this one
property = @SKIP - will not copy this property through when global @OLD is used.
property = @OLD - will copy this modified property from original object even if global @OLD is not used
property = @OLD:name - will copy [name] property from original object even if global @OLD is not used
Anything else is copied as written.
Old paths properties:
Will be used as a filter.
property = @UNSET - will apply the rule only if the property is not mapedited
"""
parser = argparse.ArgumentParser(description=desc, formatter_class=argparse.RawTextHelpFormatter)
parser.add_argument("update_source", help="update file path / line of update notation")
parser.add_argument("--map", "-m", help="path to update, defaults to all maps in maps directory")
parser.add_argument("--directory", "-d", help="path to maps directory, defaults to ../../_maps")
parser.add_argument("--inline", "-i", help="treat update source as update string instead of path", action="store_true")
parser.add_argument("--verbose", "-v", help="toggle detailed update information", action="store_true")
args = parser.parse_args()
if args.directory:
default_map_directory = args.directory
if args.map:
if args.inline:
update_map(args.map, update_string=args.update_source, verbose=args.verbose)
else:
update_map(args.map, update_file=args.update_source, verbose=args.verbose)
else:
if args.inline:
update_all_maps(update_string=args.update_source, verbose=args.verbose)
else:
update_all_maps(update_file=args.update_source, verbose=args.verbose)
@@ -0,0 +1,54 @@
# Map Conflict Fixer/Helper
The map conflict fixer is a script that can help you fix map conflicts easier and faster. Here's how it works:
## Before using
You need git for this, of course. Make sure your development branch is up to date before starting a map edit to ensure the script outputs a correct fix.
## Dictionary mode
Dictionary conflicts are the easiest to fix, you simply need to create more models to accommodate your changes and everyone elses.
When you run in this mode, if the script finishes successfully the map should be ready to be committed.
If the script fails in dictionary mode, you can run it again in full fix mode.
## Full Fix mode
When you and someone else edit the same coordinate, there is no easy way to fix the conflict. You need to get your hands dirty.
The script will mark every tile with a marker type to help you identify what needs fixing in the map editor.
After you edit and fix a marked map, you should run it through the map merger. The .backup file should be the same you used before.
## Priorities
In Full Fix mode, the script needs to know which map version has higher priority, yours or someone elses. This important so tiles with multiple area and turf types aren't created.
Your version has priority - In each conflicted coordinate, your floor type and your area type will be used Their version has priority - In each conflicted coordinate, your floor type and your area type will not be used
### IMPORTANT
This script is in a testing phase and you should not consider any output to be safe. Always verify the maps this script produced to make sure nothing is out of place.
### Map Conflict Fixer/Helper guide
1. **Check out work branch** - Checkout your own branch.
2. **Prepare Maps** - Run "Prepare Maps.bat" in the tools/mapmerge2/ directory.
3. **Pull** - Merge desired branch into your branch.
4. **Run map conflict fixer** - Run "Run Map Conflict Fixer.bat" in the tools/mapmerge2/map_conflict_fixer directory.
5. **Mode 0. Mode 1 if necessary** - Try using mode 0, if it does not work you would have to manually resolve issue using mode 1.
6. **Check manually** - Check the file manually, open map file that ends with ".fixed". For more instructions check Full Fixe Mode section bellow.
7. **Rename .fixed.dmm to be the original** - After conflicts are resolved replace original file with .fixed.dmm file. Make sure to remove .fixed.dmm into normal format(.dmm)
8. **Run mapmerge** - Run "mapmerge.bat" in the tools/mapmerge2/ directory.
#### Full Fix Mode (1)
In full fix mode you need to manually fix tiles that conflict. Luckly the tool has marked them with special map object that has path "/obj/debugging/marker"
1. **Open .fixed.dmm** - Open the map .fixed.dmm file in Dream Maker.
2. **Look for marker** - Look for the marker, the easiest way is to search it. The search panel is located on the left side in between list of objects and compile output screen. Search for "/obj/debugging/marker".
3. **Find instances** - Find all instances of marker on the map.
4. **Fix instances manually** - Once you found marker check the tile, it must contain objects from both branches(mistly duplicates). Remove undesired objects.
5. **Double check** - Make sure there are no more markers left and no duplicats on the map
6. **Save file** - Save the file in Dream Maker.
7. **Follow next steps** - Follow the steps 7 and 8 from "Map Conflict Fixer/Helper guide".
@@ -1,4 +1,4 @@
@echo off
SET RELATIVEROOT="../../"
SET RELATIVEROOT="../../../"
python map_conflict_fixer.py %1 %RELATIVEROOT%
pause
@@ -12,7 +12,7 @@ def main(relative_root):
print("--- DISCLAIMER ---")
print("This script is in a testing phase. Verify all the results yourself to make sure you got what you expected. Make sure to read the readme to learn how to use this.")
input("Press Enter to GO\n")
file_conflicts = map_helpers.run_shell_command("git diff --name-only --diff-filter=U").split("\n")
map_conflicts = [path for path in file_conflicts if path[len(path)-3::] == "dmm"]
@@ -78,11 +78,11 @@ def main(relative_root):
if mode == map_helpers.MAP_FIX_FULL:
print("After editing the marked maps, run them through the map merger!")
input("Press Enter to start.")
print(".")
time.sleep(0.3)
print(".")
for i in valid_indices:
path = map_conflicts[i]
print("{}: {}".format(ing, path))
@@ -97,7 +97,7 @@ def main(relative_root):
base_map = map_helpers.parse_map(base_map_raw_text)
if map_helpers.fix_map_git_conflicts(base_map, ours_map, theirs_map, mode, marker, priority, relative_root+path):
print("{}: {}".format(ed, path))
print("{}: {}".format(ed, path + ".fixed.dmm"))
print(".")
main(sys.argv[1])