Below is an example bash configuration that allows conda (and other similar tools) to operate in all types of bash shell, including non-interactively.

Why the #>(! won’t conda work in my script!!

…or, a tale of too many bash config files.

Shell types

So, some background. Shells, including bash, have multiple modes of invocation. The most familiar one is an interactive login shell. This is what you get when you ssh into a server, or open a TTY (Ctrl-Alt-F2 or so). Then, there’s what happens when you open a terminal while already logged in, which gets you an interactive shell. Then, then’s what happens when you run bash some-script.sh, which is a non-interactive, non-login shell.

This is all very confusing, so here’s a table summarising when you will get each form of shell.

InteractiveNon-interactive
Loginssh, Linux TTYbash -l script.sh
Non-LoginOpen a Terminalbash script.sh; qsub job.sh1

Bash startup files

So far, so arcane. Where this matters, is that various programs need their initialisation code to be run before one can use them. The package managers conda and mamba are perfect examples, though there are others. Where this interacts with the above shell modes is that each mode will source a specific configuration file. The most famous of these is ~/.bashrc, which is sourced for interactive, non-login shells. There is also ~/.bash_profile, which is sourced for interactive login shells. See the bash manual for more info on these invocation modes and their respective startup files.

Again, this will make more sense with a table

InteractiveNon-interactive
Login~/.bash_profile~/.bash_profile
Non-Login~/.bashrc??????

The missing startup file?

You’ll notice the ????? in the bottom right square corresponds to running bash as a script, or e.g. via a job scheduler like SGE or PBS1. I say ?????? as there isn’t a file path that is loaded. Instead, what happens is that bash will source the file named in the environment variable $BASH_ENV. What to do?!?

We can solve all this confusion by creating a file that I’ll name ~/.bash_env, but you could name whatever you want. This file should contain all environmental variables you would like defined in any instance of bash. This will include things like export PATH=... to include user-installed software (make sure this contains ~/.local/bin), as well as other similar variables like MANPATH, CPATH, C_INCLUDE_PATH, CPLUS_INCLUDE_PATH, LD_LIBRARY_PATH, and PKG_CONFIG_PATH. You should also include any program initialisation code needed for programs that interact closely with your shell. These include things like conda, mamba, and lmod (a.k.a module load).

Then, in your ~/.bashrc, we define export BASH_ENV=~/.bash_env, and in ~/.bash_profile we source ~/.bashrc. This ensure that we get all the environmental variables, and all programs initialised, in every shell regardless of type. Note that zsh actually formalises this, and has a hard-coded startup file to be sourced in any kind of shell named ~/.zsh_env. Much more sensible…

The solution: BASH_ENV

Below, I post examples of these config files from a server I have an account on. You should be able to copy-paste these to your homedir on most linux systems, but be careful to make backups of any existing config files, and make sure you update the paths to match your server and copy any existing important configuration from your old config files. And obviously, the conda configuration below uses paths on our cluster, adjust to your system.

Another word of caution: many systems actually already define $BASH_ENV, e.g. to enable a module system (like lmod) or something similar. In these cases, please take note of the value of $BASH_ENV before doing this setup, and make sure your ~/.bash_env sources these files (e.g. add source /usr/share/lmod/lmod/init/bash to ~/.bash_env, if $BASH_ENV evaluated to /usr/share/lmod/lmod/init/bash).

Reach out over email if you have any questions or something breaks.

Example ~/.bash_env

# BASH_ENV: for anything useful in *every* bash shell

# Update path env vars for each prefix of installed software.
INSTALL_PREFIXES=( "$HOME/.local" "/usr/lib/debian-med/")
for prefix in "${INSTALL_PREFIXES[@]}"
do
    if [ -d "$prefix/bin" ] ; then
        export PATH="$prefix/bin:$PATH"
    fi

    if [ -d "$prefix/lib" ] ; then
        export LD_LIBRARY_PATH="$prefix/lib:$LD_LIBRARY_PATH"
    fi

    if [ -d "$prefix/include" ] ; then
        export CPATH="$prefix/include:$CPATH"
        export C_INCLUDE_PATH="$prefix/include:$C_INCLUDE_PATH"
        export CPLUS_INCLUDE_PATH="$prefix/include:$CPLUS_INCLUDE_PATH"
    fi

    if [ -d "$prefix/lib/pkgconfig" ] ; then
        export PKG_CONFIG_PATH="$prefix/lib/pkgconfig:$PKG_CONFIG_PATH"    fi

    if [ -d "$prefix/share/man" ] ; then
        export MANPATH="$prefix/share/man:$MANPATH"
    fi
done

# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/usr/local/mambaforge-current/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/usr/local/mambaforge-current/etc/profile.d/conda.sh" ]; then
        . "/usr/local/mambaforge-current/etc/profile.d/conda.sh"
    else
        export PATH="/usr/local/mambaforge-current/bin:$PATH"
    fi
fi
unset __conda_setup
# <<< conda initialize <<<

Example ~/.bashrc

# BASHRC: For things mainly useful for a human

#Aliases
alias ls="ls --color=auto"
alias l="ls -lhF"
alias ll="ls -lhFa"
alias gst="git status"
alias gaa="git annex add"
alias gas="git annex sync"
alias glol="git log --oneline --graph --decorate"

# use a two-line bash prompt
PS1="$PS1\n$ "

if [ -e "$HOME/.bash_env" ]
then
    export BASH_ENV="$HOME/.bash_env"
    source "$BASH_ENV"
fi

Example ~/.bash_profile

# Make any login shell behave the same as a typical interactive shell
if [ -e ~/.bashrc ] ; then source ~/.bashrc; fi

  1. It’s rather system-dependent what kind of shell you’ll get from a cluster queuing system, but quite often you’ll do something like bash somescript.sh from within the job, in which case you end up with a non-interactive non-login shell anyway. ↩︎ ↩︎