Setting Up Run-time Environment for WRFHydro Hands-on Training v5.2.x on CyberGIS-Jupyter for Water¶

The HydroShare project is pleased to bring you this notebook that can set up a run-time environment on the CyberGIS-Jupyter for Water (CJW) platform for WRFHydro Hands-on Training v5.2.x (Nov 2020). In contrast to the Docker-based local setup, this HydroShare solution does not require installation or downloading of any software or data onto your local computer, and it enables you to access to more powerful computing resources in a clould-based CJW environment. All necessary materials required to complete this training are remotely accessible through a browser (Google Chrome recommended).

This notebook retrieves the WRFHydro model codes and relevant data from different official repos on Github and Google Drive managed by the NCAR/UCAR WRFHydro Development Team, and puts them in certain directory structure (same as the Docker-based local setup) required by the training notebooks. Specifically, three new folders will be created (wrf-hydro-training, GIS_Training, and WRF_WPS) alongside. The training notebooks are stored in wrf-hydro-training --> lessons as shown below.

alt text

The following table lists all the data (and sources) used by this notebook:

Repo/File	Link	Commit/Tag/SHA-1
wrf_hydro_nwm_public	https://github.com/NCAR/wrf_hydro_nwm_public	v5.2.0-rc1
wrf_hydro_training	https://github.com/NCAR/wrf_hydro_training	57a05ca277a1618185fb5e1646757124d3b3c49e
WrfHydroForcing	https://github.com/NCAR/WrfHydroForcing	f4ddcd760ac78245b9ed1e2a6190b02167b774de
wrf_hydro_model_tools	https://github.com/NCAR/wrf_hydro_model_tools	b70bbf455623aa1ee344a548ea5f2605e661a6f4
GIS_Training	https://github.com/mcasali/GIS_Training	97972aa1c50427d04475bfe83eb04a5dbcff23ab
croton_NY_training_example_v5.2.tar.gz	https://drive.google.com/uc?id=1_Ivc02C1WWkZcuBaSl9YePrqQE5osiCp	9e3b13cb7825caf5aa8ce5e9e5267b5144538e33
nldas_mfe_forcing.tar.gz	https://drive.google.com/uc?id=10Q-0eVakrVmFwZ27ftDDtsSHsg0YBQAT	3f7f52fee7876b797ce8dc128fbc0a169480051b
geog_conus.tar.gz	https://drive.google.com/uc?id=1X71fdaSEJ5GWyNY2MDIy9cC6E7A0kihl	5f7cbbc05a4665667ff32de7fb2393fd79ec5c30

To run this notebook: Cell (top menu) --> Run All

Note: You are supposed to run this notebook only once. Each run will set up a fresh new environment, and any pre-existing training setup (codes, notebooks and model data) and your edits will be removed.

After running this notebook successfully, the lesson notebooks will have been downloaded into the workspace and you can start the training at Lesson 1, or view all notebooks here. **Until this is done these links are inactive.**

Troubleshooting: If any error message occurred in this notebook, it might be caused by some previously executed training notebooks still running and referencing model data or files on disk . Try one of the two actions below, and run this notebook again.

Stop all other training notebooks: Go to each notebook --> Kernel (top menu) --> Shutdown;
Restart Jupyter server: Control Panel (upper-righer corner) --> Stop My Server --> Start My Server;

%%bash
rm -rf ./wrf-hydro-training 2>/dev/null 
rm -rf ./WRF_WPS 2>/dev/null 
rm -rf ./GIS_Training 2>/dev/null 
rm -rf ~/wrf-hydro-training 2>/dev/null 
rm -rf ~/WRF_WPS 2>/dev/null 
rm -rf ~/GIS_Training 2>/dev/null 
rm -rf /home/docker/wrf-hydro-training 2>/dev/null 
rm -rf /home/docker/WRF_WPS 2>/dev/null 
rm -rf /home/docker/GIS_Training 2>/dev/null

! ls $(pwd)

%%bash
mkdir -p ./wrf-hydro-training
mkdir -p ./WRF_WPS
if [ -d "/WRF_WPS/WPS" ] 
then
    ln -sf /WRF_WPS/WPS $(pwd)/WRF_WPS/WPS
else
    ln -sf $EBROOTWPS/WPS-$EBVERSIONWPS $(pwd)/WRF_WPS/WPS
    echo "Using WPS from easybuild...."
fi

ln -sf $(pwd)/wrf-hydro-training ~/wrf-hydro-training
ln -sf $(pwd)/GIS_Training ~/GIS_Training
ln -sf $(pwd)/wrf-hydro-training /home/docker/wrf-hydro-training
ln -sf $(pwd)/GIS_Training /home/docker/GIS_Training
ln -sf $(pwd)/WRF_WPS ~/WRF_WPS
ln -sf $(pwd)/WRF_WPS /home/docker/WRF_WPS

# WRFHYDRO version on https://github.com/NCAR/wrf_hydro_nwm_public
wrfhydro_version="v5.2.0-rc1"
# wrf_hydro_training branch or commit on https://github.com/NCAR/wrf_hydro_training
training_version="57a05ca277a1618185fb5e1646757124d3b3c49e"
# forcing on https://github.com/NCAR/WrfHydroForcing
forcing_version = "f4ddcd760ac78245b9ed1e2a6190b02167b774de"
# model tools version on https://github.com/NCAR/wrf_hydro_model_tools
model_tools_version = "b70bbf455623aa1ee344a548ea5f2605e661a6f4"
# gis training version on https://github.com/mcasali/GIS_Training
gis_version = "97972aa1c50427d04475bfe83eb04a5dbcff23ab"

!wget https://github.com/NCAR/wrf_hydro_nwm_public/archive/{wrfhydro_version}.tar.gz
!tar -xzvf {wrfhydro_version}.tar.gz
!rm {wrfhydro_version}.tar.gz
!mv ./wrf_hydro_nwm_public* ./wrf-hydro-training/wrf_hydro_nwm_public

!git clone https://github.com/NCAR/wrf_hydro_training
!cd ./wrf_hydro_training && git checkout {training_version}
!mv ./wrf_hydro_training/lessons/training ./wrf-hydro-training/lessons
!rm -rf ./wrf_hydro_training

import sys
!{sys.executable} -m pip install nbformat

import nbformat
import glob

# Set notebooks to the WRFHydro kernel on CJW
for fp in glob.glob("./wrf-hydro-training/lessons/*.ipynb"):
    nb = nbformat.read(fp, as_version=4)
    if nb.metadata.kernelspec.display_name == "Python 3" and nb.metadata.kernelspec.name == "python3":
        print("changing kernal for {}".format(fp))
        nb.metadata.kernelspec.display_name = "WRFHydro-2021-09"
        nb.metadata.kernelspec.name = "wrfhydro-2021-09"
        nbformat.write(nb, fp, version=nbformat.NO_CONVERT)
    elif nb.metadata.kernelspec.display_name == "Bash" and nb.metadata.kernelspec.name == "bash":
        print("changing kernal for {}".format(fp))
        nb.metadata.kernelspec.display_name = "Bash-2021-09"
        nb.metadata.kernelspec.name = "bash-2021-09"
        nbformat.write(nb, fp, version=nbformat.NO_CONVERT)

!pip install gdown

%%bash
gdown https://drive.google.com/uc?id=1_Ivc02C1WWkZcuBaSl9YePrqQE5osiCp
tar -xzvf croton*.tar.gz
rm croton*.tar.gz
mv ./example_case ./wrf-hydro-training/example_case

!git clone https://github.com/NCAR/WrfHydroForcing.git
!cd ./WrfHydroForcing && git checkout {forcing_version}
!mv ./WrfHydroForcing ./wrf-hydro-training/WrfHydroForcing

!git clone https://github.com/NCAR/wrf_hydro_model_tools.git
!cd wrf_hydro_model_tools && git checkout {model_tools_version}    
!mv ./wrf_hydro_model_tools ./wrf-hydro-training/wrf_hydro_model_tools

%%bash
gdown https://drive.google.com/uc?id=10Q-0eVakrVmFwZ27ftDDtsSHsg0YBQAT
mkdir -p ./wrf-hydro-training/regridding
mv nldas*.tar.gz ./wrf-hydro-training/regridding/nldas_mfe_forcing.tar.gz

%%bash
gdown https://drive.google.com/uc?id=1X71fdaSEJ5GWyNY2MDIy9cC6E7A0kihl 
tar -xzvf geog_conus.tar.gz
rm geog_conus.tar.gz
mv ./geog_conus ./WRF_WPS/geog_conus

!git clone https://github.com/mcasali/GIS_Training
!cd GIS_Training && git checkout {gis_version}

print("Done")

Next - Start from Lesson 1 or view All Lessons ¶

If there was any error occurred in this notebook, you can try to re-run it by: Kernel (top menu) --> Restart & Run All. If the same error persisits, you can try restarting the jupyter environment: Control Panel (upper-right corner) --> Stop My Server --> Start My Server, and then run this notebook again.

Setting Up Run-time Environment for WRFHydro Hands-on Training v5.2.x on CyberGIS-Jupyter for Water¶

Next - Start from Lesson 1 or view All Lessons¶

Next - Start from Lesson 1 or view All Lessons ¶