Getting Started with DES at Fermilab

Accounts and Access

To access Fermilab's computing resources and the DES Collaboration tools follow the instructions found here:
https://data.darkenergysurvey.org/fnalmisc/onboard/onboard.html

By following these instructions you should be granted: (1) a Fermilab user name, (2) a Fermilab Kerberos account, (3) an entry in the DES membership database, (4) a request for access to the DES data through DESDM, and (5) access to the DES Collaboration wiki on Redmine. Once you have access to Redmine, more complete documentation for Collaboration tools and services can be found here:
https://cdcvs.fnal.gov/redmine/projects/desecs/wiki/Getting_Started

After you have been granted a Fermilab username, you can follow the instructions here to get an account on the DES cluster machines. This may involve sending an email to Liz Buckley-Geer and Brian Yanny, so be sure to include some context regarding who you are and what you will be working on. You can find out more about the DES computing resources at Fermilab here. Once you have been granted access to the DES cluster, you can login to one of the DES machines using kerberos and ssh:

> kinit ${USER}@FNAL.GOV
> ssh -XY ${USER}@des80.fnal.gov

There are some quirks for getting ssh to forward the right credentials, so check out the instructions on the "Getting Started" page if you have trouble. There are also several pages associated with Fermilab kerberos access and troubleshooting. For X11 window forwarding on Mac OS, you will need XQuartz.

While at Fermilab, there are several wireless networks available: fgz, guest, eduroam. The guest network is accessible to everyone, but has limited bandwith and may not have all ports open. The eduroam network should be accessible if you are member of an eduroam institution, but also has limited bandwith. The fgz network is the primary Fermilab wireless network. Devices can be registered for the fgz network by following these instructions.

Using Linux

Most scientific computing these days is performed on some flavor of the Linux operating system. Over the past few years, the entry barrier to Linux has be significantly lowered by the popularization of user-oriented Linux operating systems such as Ubuntu. Fermilab computers run Scientific Linux which is a rebuild of Red Hat Enterprise Linux. There are various Linux and command line tutorials spread accross the web. Here are a few that caught my eye:

While these tutorials may be a good place to start, Google is the single most important resource when learning Linux (and programming in general). A close second is the Stack Overflow user forum. However, it can be difficult to find the answer to a question that you don't know how to ask. If that is the case, find a person to ask!

Scientific Python with Anaconda

Python has become the standard for scientific computing in astrophysics and cosmology. One straightforward and robust way to install python and its suite of scientific computing packages is through Anaconda. Anaconda provides an installation of python, numpy, scipy, matplotlib, etc. as well as a package manager, conda, for installing other commonly used packages like astropy.

If you are working on the DES cluster machines at Fermilab, Anaconda and many of the common packages used for DES science are already installed, and you can skip to the "Setup an Environment" section. If you find that a package you need is not already installed, Anaconda is flexible enough to allow you to easily install and integrate that package into a new python environment (see "Create an Environment"). Finally, if you want to install Anaconda on your own machine, some useful links are included in the "Installation" section.

Setup an Environment

The DES cluster has a centralized Anaconda installation. The centralized installation contains many common packages and should allow you to get started quickly. Below we provide a minimal set of instructions to get you started; more detailed instructions can be found on the conda documentation.

To get started you need to source the conda setup script (there is an equivalent conda.csh for cshell users): 

> export CONDA_DIR=/cvmfs/des.opensciencegrid.org/fnal/anaconda2
> source $CONDA_DIR/etc/profile.d/conda.sh

You can then activate the base environment of the anaconda install with: 

> conda activate base
This environemnt contains the full suite of Anaconda Python 2.7 packages; however, it is not explicitly oriented towards astrophysics/cosmology. For astrophysics/cosmology applications there is a DES environment (des18a) that can be activated from the centralized Anaconda installation: 
> conda activate des18a
There is also a newer (Python 3.7) DES environment that can be activated with: 
> conda activate des20a
More information on the DES conda environments can be found here. After setting up conda, you can list available environments with: 
> conda env list
By default, conda will search for environments under the root installation (i.e., $CONDA_DIR/envs), your home directory (i.e., $HOME/.conda/envs), and anything that you have listed in $HOME/.conda/environments.txt. You can modify your conda search path either by setting the CONDA_ENVS_PATH environment variable or appending to the envs_dir entry in your .condarc file. A more detailed discusion of conda configuration can be found here with a detailed discussion of conda directory paths here.

Run a Jupyter Notebook

It is possible to run a jupyter notebook on the DES cluster, but view the notebook from your local webbrowser. Following the instructions here, start the notebook server on the remote machine (i.e., des80) 
> ssh ${USER}@des80.fnal.gov
> conda activate des18a
> jupyter notebook --no-browser --port=8889
then connect to this server from an ssh tunnel on your local machine: 
> ssh -N -L localhost:8888:localhost:8889 ${USER}@des80.fnal.gov
Then direct your web browser to localhost:8888.

Create an Environment

At some point it may be necessary to update packages or install new packages that are not included in the centralized installation. Additionally, you may want to create your own custom software environment associated to a specific analysis. To achieve these goals, you will want to create a new conda environment. By default, new environments that you create will live in your home directory under $HOME/.conda. Since conda environments can grow quite large, the best practice is to symlink $HOME/.conda to one of the large DES disks before you start creating environments: 

> mkdir -p /data/des80.a/data/${USER}/conda
> ln -s /data/des80.a/data/${USER}/conda ${HOME}/.conda
To create a new environment, start by performing the general conda setup described. Use the conda create command to create a new environment containing numpy, scipy, and matplotlib: 
> conda create --name myenv numpy scipy matplotlib
This command will create an environment named myenv in $HOME/.conda/envs. Your new environment can be activated similarly to the centralized environments (see above for directions on modifying your environment path): 
> conda activate myenv
If you find yourself making a bunch of environments, make sure to clean up your conda directory from time to time: 
> conda clean --packages
> conda clean --tarballs

Installation

Anaconda is already installed on the Fermilab DES cluster and a personal installation should not be necessary (skip to the next section for setup instructions). However, if you want to install Ananconda on your own machine, follow the installation instructions here:
https://docs.continuum.io/anaconda/install

Once Anaconda is installed, you can use conda to install additional packages into working environments. For more details on package installation see here:
http://conda.pydata.org/docs/using/using.html

For the more advanced user, it is also possible to perform a light weight install of conda itself using the Miniconda:
http://conda.pydata.org/docs/install/quick.html

Python Tutorials

Here are the official tutorials for Python 2 and 3:
https://docs.python.org/2/tutorial/
https://docs.python.org/3/tutorial/
As a more interactive introduction, I've heard some good things about Code Academy: https://www.codecademy.com/learn/python

The main packages that are used within DES for scientific computing are numpy, scipy, matplotlib, and astropy. Below I list some of the official tutorials, but I'm sure there are better tutorials out there.