How Readily Accessible Developer Tools Can Serve the DIVERSE Project and Forest Ecology

Ecologists increasingly need to deal with convoluted computer models and scripts, with numerous dependencies, without a background in computer science. When these models are difficult to understand and run, then the results are not replicable, sometimes even not reproduceable, especially in the long run. However, with a combination of well-known computer tools like Docker, Jupyter Notebook and GitHub (more on this later), we can ensure that our research in DIVERSE remains replicable and re-usable in the future. 

Developing large-scale models to simulate complex systems is arduous to say the least. In ecology, the world of modeling faces three important challenges: 

• The reproducibility issue when it comes to complex models and simulations 

• The ‘It works on my machine’ dilemma 

• The long-term use of the models 

The good news? These challenges are surmountable when ecologists knock on developers’ doors for help. 

Before we dive in, here is the story of how our research team found itself face to face with the issue of replicability. I work for the DIVERSE project, a Pan-Canadian research initiative dedicated to advancing forest management. DIVERSE is using LANDIS-II, a cutting-edge forest landscape model, to assess the long term and large-scale impacts of different forest management strategies on Canadian forests. The calibration of this model is very complex. It relies on many extensions that the user can choose, each requiring dozens to hundreds of parameters. These parameters need to be calibrated so that the outputs of each extension are realistic. As such, decisions are made at each step of the calibration of each extension based on the data in the literature and on the outputs observed from the model, with many iterative rounds. 

For example, in DIVERSE, we are working with the PnET extension for LANDIS-II, which simulates tree growth based on light and water competition between trees. If the outputs from one of our PnET simulations show that the maple saplings in a forest will grow to 5 meters in one year [way too much growth], that tells us that our calibrations have probably gone wrong.  

As such, calibrating LANDIS-II for simulations can take months of work. It is not an easy endeavor. 

Up to now, ecologists have had difficulties keeping track of the myriad of decisions required for model calibration. Calibration procedures thus tend to leave a trail of various files filled with numbers, equations, graphs and small notes that don’t even make sense to their own authors after a couple of months. In this context, if a researcher wants to replicate a study published 5 years ago, chances are that its authors will no longer recall the specifics of their model calibration. This leaves little room to criticize, improve, and especially re-use calibration procedures or the parameters they produced. 

If reproducibility is key to good science and yet we can’t reproduce the simulations we do because of a lack of information about our calibrations, then what is the point? 

One tool that we have started using within DIVERSE to address this challenge and create a record of our calibration process is Jupyter Notebook. This interface has become quite popular because it is convenient, allowing users to write Python code and execute it in cells while also writing textual documentation with it at the same time, all by using their web browser. The resulting pages are a dance between pieces of text, codes and outputs, which goes beyond keeping a script. This could be compared to a recipe book handmade by your grandmother, with recipes cut from a journal and handwritten notes of modifications she made to the original recipe, and what it tasted like after. Jupyter becomes a “recipe book” with traces and notes of what has been done previously in the model, with clear explanations and justifications. 

DIVERSE researchers can use Jupyter Notebook to document the calibration of their LANDIS-II model. Great! But this is only the first step to ensure replicability and collaboration. The second step is to make sure that all LANDIS-II extensions and the Jupyter Notebook can work together and be used on different computers by all our researchers, because believe us, you don’t want to do this on your own. 

You may be wondering – couldn’t we just export our files to send them to a coworker who needs them? Unfortunately, it is not that simple because of a phenomenon that is infamously called dependency hell.  To understand this issue better, let’s go back to your grandma’s handmade recipe book analogy.  

Normally, one should be able to make their grandma’s recipes from her handmade book and have them taste exactly like how she made them. But what if some ingredients don’t exist anymore? What if brands have changed? Or what if you don’t have the same cooking instruments that your grandma was using? All these things would make it harder to reproduce the recipe, and you would need to make your own adjustments to accommodate. 

It is the same thing with models: to reproduce their outputs, we need to use the same ingredients and tools as when the models were developed in their virtual environment. Put simply, each model is dependent on its virtual environment. The challenge is that these virtual environments (R packages, Python scripts, etc.) are updated regularly, which is necessary, but can create errors when trying to run the model. This can also happen when running the model on a different computer, since each computer has its own unique virtual environment and could easily have different functions from other devices.  

These dependencies quickly become an issue as we do research with these models: either other scientists can’t readily run a model on their own computer, or the model might not even be able to run in the future. For people working in software development, there is even a meme to describe this very problem: “But it works on my machine!”.  

To avoid this pitfall, we have started to use Jupyter Notebook inside a platform called Docker. Docker creates virtual machines – it essentially simulates a computer within a computer.  

When using docker, you can create a docker image, a blueprint of the virtual machine it created. This simple set of instructions can be opened in text files and sent to fellow co-workers, reviewers, or collaborators to allow them to re-create your virtual machine on their own computer and run your model without hurdles. It’s like shipping your computer along with your model.  

Why would you want to do that? Because in this way, you can create the virtual environment needed to run your model and freeze it, so it does not change; it stays outdated. Your model and its dependencies will stay the same and still be usable 10 years from now. Who knew being outdated could be that great? 

With these two innovations together, Jupyter Notebook and Docker in tandem, forest ecologists could make a lot of progress in terms of sharing their models, calibration methods, and approaches. The last step missing is the collaborative approach of science: having many scientists from the same field working towards improving existing models and calibrations rather than reinventing the wheel each time. 

To catalyze scientific collaboration around the LANDIS-II model, the DIVERSE team has created a GitHub repository for our docker image containing the calibrations of our model. GitHub is an open access platform where everybody can download the files to modify and use them for their own projects. People can also access and edit the codes shared in the Git Versioning system, which is a robust system where all modifications of the codes are registered and associated with a user, making the collaborative process transparent.  

Lastly, GitHub has the system GitHub pages that makes it free and easy to distribute the Jupyter Notebook as a web page. DIVERSE has shared all its Jupyter Notebook files containing the calibration approach of LANDIS-II on GitHub pages, making the calibration procedure easy to read without even having to install Docker or run the whole model on your own computer. 

So now, instead of fighting with their own models that can’t easily be reproduced, nor used by other research teams, ecologists can work collaboratively on complex models and make them better together. 

In summary, the three innovations we share here are a step in the right direction in forest ecology, that could probably be extended to all fields of ecology and other spheres in science.

• Jupyter Notebook to show the process from the calibration to the simulations of the models; 

• Docker to share the model and its calibration with others; 

• GitHub to work collaboratively and ensure the prosperity of the models we spend years to develop. 

While these are the tools we use in DIVERSE, there are many other possibilities to learn about in programs such as Data Carpentry.

In brief, computers are very complex, and while we feel like they have always been part of our lives because they are ubiquitous in today’s world, they are still quite new to us in the grand scheme of science. We need to adapt our ways of working in ecology to make sure that the research we do respects the guiding principles of science, namely reproducibility and replicability.