Author: Alexander Michels
This notebook is a very brief explanation of what Lmod is and how to use it on CyberGISX.
Lmod is an implementation of Environment Modules, much of what is said here is true for any environment modules system but there are many features which are unique to Lmod. Environment Modules provide a convenient way to dynamically change the users’ environment through modulefiles. This includes easily adding or removing directories to the PATH environment variable.
Basically, Lmod is software that allows us have multiple co-existing versions of software and simply "load" and "unload" them at will!
While Lmod does provide a command line interface (CLI), we recommend you DO NOT use Jupyter ! commands to interact with Lmod because it won't display outputs correctly and often will not accomplish what you are hoping to. Lmod works by manipulating environmental variables like PATH and ! commands spawn subprocesses so your environment will not "remember" what happened in them.
As an example of this "forgetfulness" of ! let's try to set out name as an environmental variable and then in the same cell try to print the variable:
!export MYNAME="Alex"
!echo $MYNAME
Similarly, a module load xxx won't be remembered if run through an ! command.
Luckily, Lmod comes with a very helpful Python package to interact with it! Some of the details can be found in our Knowledge Base and the implementation is very simple if you want to check it out.
Basically after we load it, instead of running !module list we will run module("list"). More details on the module command syntax can be found here. Let's load the Python package and try it out:
import sys, os # we need this to add the $MODULESHOME path to our Python path
sys.path.insert(0, os.path.join(os.environ['MODULESHOME'], "init")) # add to the python path
from env_modules_python import module # import package
output = module("list") # the output is the return code and stderr output, we catch it to make the notebook cleaner
The list command displays the currently loaded modules. To view the available modules, we can use avail:
output = module("avail") # the output is the return code and stderr output, we catch it to make the notebook cleaner
Lastly, we are going to try to load a different version of GDAL and demonstrate that it is in fact a new version. First, let's check the version we have:
!gdalinfo --version
Currently, we are using the GDAL version 3.3.1 from the Anaconda environment, but suppose we need to use another version available through Lmod like 3.6.2 (see GDAL/3.6.2-foss-2022b in the avail output). To use that, we simply:
output = module("load", "GDAL/3.6.2-foss-2022b")
The output tells us that many packages have changed versions to accomdate this switch!
To verify, let's check the version again:
!gdalinfo --version
We have a paper under submission about the architecture of CyberGISX, but in the meantime check out: