Testing your mod

There are several mods that can help us test our own mods, besides using debugtools. Usually we'll create our own test files inside of them, there's no need to make a separate mod with mixintos just for testing your custom mods.

Microworld
Stonehearth tests
- Setup stonehearth_tests
Stonehearth autotests
- How to create your own custom autotests
Lua profiler scripts

Microworld

The microworld mod helps us to test our mod faster, by creating a custom test world where we can use debug tools to spawn our modded entities, without having to go through the embarkation process every time we want to test something.

For testing biomes, however, there's no easy way, since you will want to check the whole map generation, not just a flat tiny world.

Setup Microworld

The microworld mod comes bundled with the game now (you can also find it on Github).

Go to the Mods menu from the main menu, search for it in the Local mods column, and enable the "Microworld for Tests" mod.
Close the game.
Edit your user_settings.json file.
- If you installed the game with the Humble Bundle installer, it will normally be at C:\Program Files (x86)\Stonehearth.
- If you have installed it via Steam, then it will be at C:\Program Files (x86)\Steam\steamapps\common\Stonehearth.
Some of these keys might already exist, in that case add only what's missing.
```
{
   "user_id": "your_user_id",
   "game" : {
      "main_mod" : "microworld"
   },
   "mods" : {
      "microworld" : {
         "world" : "harvest_test"
      }
   }
}
```
The "main_mod" key tells the game which mod should be loaded as the main mod. In this case, we'll load the Microworld mod first, which will skip the logo / main menu screens and will load directly a small world. Make sure to invalidate this key whenever you want to play the game again ("_main_mod" : "microworld"), or use the Esc menu to go back to the Main menu or load a savefile.

The "world" option inside the settings for the microworld mod points to "harvest_test" in this case (by default it will run the mini_game world). All the test worlds, located inside the microworld/worlds directory, end their names by _world.lua. You must follow that naming convention in order for the mod to run the test. For instance, if your test world is called mytest_world.lua, in the user_settings you will specify it as "mytest".
Finally, start the game.

You can also run this mod from the command line (make sure that the mod is enabled first!):

  Stonehearth.exe --game.main_mod=microworld --mods.microworld.world=harvest_test

Default worlds

There exist 4 test worlds in the microworld mod (at the time this page was written): data_driven_world.lua, harvest_test_world.lua, mini_game_world.lua and settlement_test_world.lua.

You can try changing your user_settings to see what each of them has. For the data_driven world, you'll need to add an extra key:

  "mods" : {
     "microworld" : {
        "world" : "data_driven",
        "data_driven_index" : "microworld:data_driven:world:profession_test"
     }
  }

The "data_driven_index" key points to an alias defined in the microworld manifest. You can add more aliases there for your custom worlds.

How to create your own custom worlds

You can find many functions to create objects in your custom worlds inside microworld/micro_world.lua, and you can also use code from the game or other mods.

The data_driven_world.lua world uses JSON data from the data/data_driven_world subfolder in order to generate the custom world. You can try that approach if you don't want to use Lua.

As explained above, you need to add an alias to the microworld's manifest to add your data driven world, use the "data_driven" world in your user_settings together with the "data_driven_index" of your custom JSON world.

In the JSON files you can specify the size of the world, the citizens you want to spawn and where, items to spawn and where, chunks of terrain and where, etc. Take a look at the examples inside data/data_driven_world, and at the comments inside data_driven_world.lua.
The rest of worlds are written in Lua. This is the minimum code needed to create a micro world:
```
local MicroWorld = require 'micro_world'
local SettlementTest = class(MicroWorld)

function SettlementTest:__init()
   self[MicroWorld]:__init(100)
   self:create_world()
   -- More code
end

return SettlementTest
```
We require the micro_world.lua file in the first line, which contains the Microworld class. Then we instantiate the Microworld class, and define an __init() function for our object to initialize our custom world. Inside it, we call the initialization function from the Microworld class passing the diameter (in number of blocks) that we want for our test world, and call the create_world() function (also from the Microworld class) to actually create the flat terrain. Finally, we return our object.

We can create more functions in our class if we need to. There are already many functions in micro_world.lua that are useful to prepare a test world, you can call them like we called the create_world() function, and pass any parameters that you need. For example, the create_world() function can receive a kingdom URI and a biome URI as arguments, to generate the test world with them.

Note that microworld attempts to simulate a regular game but it does not actually go through the same game flow as a regular game. For example, we have to manually initialize the services we want to use in MicroWorld:create_world. So if you're running into a bug that only happens in tests, it may be due to the tests themselves as opposed to the game. Generally the tests should match the game behavior but it isn't guaranteed.

Stonehearth tests

These tests are pretty similar to the Microworld worlds explained above. The stonehearth_tests mod makes use of the microworld mod to generate test worlds, so you'll need to enable both in the Mods menu if you plan to use the stonehearth_tests mod.

The development team used this mod to manually test features in isolation as they were being developed, or to try reproducing bugs in small environments.

You have plenty of examples to reuse in this mod, so if you're new to coding, you can fiddle with Lua here until you feel comfortable enough to attempt more complex things in your mod.

icon The same warning stated above for microworld can also be applied for the stonehearth_tests mod. For example, when running tests you'll notice that the game master service hasn't been initialized (the Campaign Browser debug tool looks blank when you open it - thankfully it has a button to start it from the UI). So you'll need to initialize them manually. Also, multiplayer doesn't work unless you do the things listed in multiplayer_test.lua.

Setup stonehearth_tests

The stonehearth_tests mod now comes bundled with the game.

Go to the Mods menu from the main menu, find the "Microworld for Tests" and "Stonehearth Unit Tests" mods in the Local mods column, and enable both of them.
Close the game.
Edit your user_settings.json file.

Some of these keys might already exist, in that case add only what's missing.
```
{
   "user_id": "your_user_id",
   "game" : {
      "main_mod" : "stonehearth_tests"
   },
   "mods" : {
      "stonehearth_tests" : {
         "test" : "pet_test"
      }
   }
}
```
The "main_mod" key tells the game which mod should be loaded as the main mod. In this case, we'll load the stonehearth_tests mod first, which will skip the logo / main menu screens and will load directly a small test world. Make sure to invalidate this key whenever you want to play the game again ("_main_mod" : "stonehearth_tests"), or use the Esc menu to go back to the Main menu or load a savefile.

The "test" option inside the settings for the stonehearth_tests mod points to "pet_test" in this case. When you create your own custom test world you can change it for the name of your Lua file. All the tests, located inside the stonehearth_tests directory, end their names by _test.lua by convention. Unlike with the Microworld mod, we do have to declare the whole name of the file (minus the extension) in the "test" field in the user_settings.json, like in the example above.
Finally, start the game.

You can also run this mod from the command line (make sure that both this and the Microworld mods are enabled first!):

  Stonehearth.exe --game.main_mod=stonehearth_tests --mods.stonehearth_tests.test=pet_test

Stonehearth autotests

We can create automated tests that will compare expected results with actual results so that we don't have to test features manually. They're useful to verify that regressions weren't introduced after changing some code.

icon Keep in mind that we may not be able to automate some things, and that issues that an autotest reproduces sometimes are not reproducible in the game and viceversa. Try to avoid flaky tests that sometimes pass and sometimes fail.

To create our own automated tests, we have two mods that now come bundled with the game. The stonehearth_autotest mod requires the autotest_framework mod in order to work, so you'll have to enable both in your user_settings.json if you plan to run autotests:

Go to the Mods menu from the main menu, search for the "Autotest Framework" and "Stonehearth Autotests" mods in the Local mods column, and enable both of them.
Close the game.
Edit your user_settings.json file.

Some of these keys might already exist, in that case add only what's missing.
```
{
   "user_id": "your_user_id",
   "game" : {
      "main_mod" : "stonehearth_autotest"
   },
   "mods" : {
      "stonehearth_autotest" : {
         "options" : {
            "run_forever" : false,
            "num_times" : 10,
            "exit_on_complete" : false,
            "continue_on_failure" : true,
            "pause_on_error" : true,
            "script" : "/stonehearth_autotest/tests/end_to_end/harvest_autotests.lua",
            "function" : "long_*",
            "group" : "nightly"
         }
      }
   }
}
```
The "main_mod" key tells the game which mod should be loaded as the main mod. In this case, we'll load the stonehearth_autotest mod first (careful when typing, it doesn't end by an 's'). Make sure to invalidate this key whenever you want to play the game again ("_main_mod" : "_stonehearth_autotest"), or use the Esc menu to go back to the Main menu or load a savefile.

Then we have several "options" that we can use:
- "run_forever" -- optional. If we set to true, it will run the specified tests forever until we close the game.
- "num_times" -- optional. Run each test a specified number of times.
- "exit_on_complete" -- optional. When false, will leave the game open after the last test finishes, instead of automatically closing the game.
- "continue_on_failure" -- optional. When true, tests will continue executing even if one of them failed.
- "pause_on_error" -- optional. When true, will pause the execution when a test fails, to be able to check the state of the game.
- "script", "function" and "group" -- with these we define which autotests we want to run. If we don't specify a script nor a group, the group called "all" will be executed. To find the available groups, look in the "groups" keys from stonehearth_autotest/tests/index.json.
  
  We can run a single script instead of a group of scripts. In that case, specify the whole path to the Lua file containing the autotests in the "script" option. By convention, we end our autotests files with _autotests.lua.
  
  We can also run a specific test from a script. To do that, define which script contains the test in the "script" option, and then write the name of the test in the "function" option. For example:
```
"script" : "/stonehearth_autotest/tests/end_to_end/harvest_autotests.lua",
"function" : "clear_test"
```
  We can also use a couple of pseudo regular expressions for the "function" option. For example, this will run all tests from death_autotests.lua that begin with regression_ :
```
"script" : "/stonehearth_autotest/tests/end_to_end/death_autotests.lua",
"function" : "regression_*"
```
  And this will run all the tests from death_autotests.lua that don't begin by regression_ :
```
"script" : "/stonehearth_autotest/tests/end_to_end/death_autotests.lua",
"function" : "!regression_*"
```
  Notice the exclamation point, and the asterisk.
Finally, start the game. Depending on your options, the window will / won't be closed automatically after the tests finish running. Also, you might be interested in increasing the logging level for the autotest_framework mod. The framework will print which scripts and tests are being executed, (and also which tests failed, together with a message) in the stonehearth.log. Tests that pass won't print any special message when they finish.

You can also run this mod from the command line (make sure that both the autotest_framework and stonehearth_autotest mods are enabled first!):

  Stonehearth.exe --game.main_mod=stonehearth_autotest --mods.stonehearth_autotest.options.script=stonehearth_autotest/tests/end_to_end/harvest_autotests.lua

How to create your own custom autotests

Most of the autotests are located inside stonehearth_autotest/tests/end_to_end, you can take a look at those to see how they work. Autotests are run at speed 4 by default, many of them rely on the game being that fast. Unlike with the microworld mod, we don't need to manually create the terrain, a default size will be used.

In general, follow these steps:

Create a Lua file inside the (e.g.) stonehearth_autotest/tests/end_to_end folder. For example my_mod_autotests.lua.
Add your Lua file to the "scripts" array inside a group from the stonehearth_autotest/tests/index.json file.
Edit your Lua file. Initially it will look something like this:
```
local my_tests = {}

function my_tests.verify_this_and_that(autotest)

   autotest:success()
end

return my_tests
```
All the tests that you add must call autotest:success() when the success condition is met, otherwise they'll fail. Then, you can make them fail if a failing condition is met by calling autotest:fail with a descriptive message of why it failed:
```
autotest:fail('failed to complete this thing')
```
In general, autotests will have the following structure:
- They will first set up a scenario by creating some entities.
- Then start listening for events that signal the success (sometimes the failure too) of the test.
- Then start some tasks by clicking in the UI or fiddling with internal values in Lua, in order to produce the success condition.
- Then wait for a while to give the game time to execute the above.
- Then fail (a timeout that will only be reached if the success condition wasn't met on time).
For example:
```
-- Verify that a hearthling can die when guts reach 0 and that the tombstone is generated
function death_tests.memorialize_death(autotest)
   local worker = autotest.env:create_person(5, 5, { job = 'worker', resources = { health = 0, guts = 8} })

   autotest.util:listen(radiant, 'radiant:entity:post_create', function (e)
         if e.entity:get_uri() == 'stonehearth:tombstone' then
            autotest:success()
         end
      end)

   autotest:sleep(5000)
   autotest:fail('failed to die')
end
```
In order to find the helper functions used in the autotests, you'll need to take a look at the autotest_framework files. For instance, in the example above we call autotest.env.create_person(), which is a function from autotest_framework/lib/server/autotest_environment.lua.

The function autotest.util.listen() is a wrapper function from autotest_framework/lib/server/autotest_util_server.lua (you can find other helper functions there, such as fail_if_expired(timeout)).

Any time you listen to an event in your autotests you must use one of the wrappers defined there, so that the autotest framework can clean them up correctly no matter if the tests passes or fails (otherwise, some listeners might still exist when the next test is executing, and might fire at the wrong time, throwing errors).

The autotest:success(), autotest:fail() and autotest:sleep() functions are defined in autotest_framework/lib/server/autotest_instance.lua, which controls the actual execution of the autotests. The parameter to the sleep() function is a number representing milliseconds.

Other common functions used in the tests are located in autotest_framework/lib/server/autotest_ui_server.lua. And of course, you can use code from the game (like retrieving components, etc).

Be careful since some things might still remain from test to test if you don't reset / clean them up, such as open UI menus.

Lua profiler scripts

With these scripts you can profile your mod's Lua performance. You can find them here.

Setup instructions:

Place the following in your user_settings.json:

"lua" : {
    "enable_cpu_profiler" : true,
    "enable_memory_profiler" : true,
    "cpu_profiler_method" : "sampling", // or "time_accumulation"
    "profiler_instruction_sampling_rate" : 1,
    "max_profile_length": 10000 //optional, how long in ms we can run the profiler
},
"simulation" : {
    "initial_speed_override" : 0,
    "long_profile_tick_threshold" : 500
}

Remove the comments after pasting into your user settings, and adjust the config values as needed.

Build the lua file map by navigating to the folder where you downloaded the lua_profiler scripts folder to, and then running the Python script with the following arguments:
```
collect_lua_file_map.py [PATH_TO_MODS_FOLDER] lua_file_map.js
```
Replace [PATH_TO_MODS_FOLDER] with the path to your stonehearth mods folder.
Load up the game you want to profile.
Wait for the UI to come up, open the Performance Monitor debug tool.
When ready (after clicking the speed 1 button and waiting for initial script catch up), click on the profiling button (the play button). If you are looking for what is causing sudden hitches, check the Long Ticks Only checkbox.
Press the stop button to finish profiling.
Open Chrome and navigate to the lua_profiler.html file in the lua_profiler folder.
Click on the "Choose Files" button and navigate to the profile dump under Stonehearth\profiler_output\DATE_AND_TIME_OF_PROFILE_CAPTURE. The profiler_output folder can be found in the same folder as the mods folder.
Select ALL files under that folder and click open.
When the profiler finishes loading the files, it will populate the rows with each method name, total time, # calls, percentage of lua consumption (this number is the important one), and the file path.
Look at the function that is taking the most amount of time and figure out if there's a way to make it faster.
Profile again!

icon IMPORTANT: launch a new tab of lua_profiler/lua_profiler.html every time you need to load a new profile.

Note: If you are getting a game crash while profiling in ai::CreateAction, add a check for ai.get() on the preceding line.