Below is an example bash configuration that allows conda
(and other similar tools) to operate in all types of bash shell, including non-interactively.
Why the #>(! won’t conda work in my script!!
…or, a tale of too many bash config files.
Shell types
So, some background. Shells, including bash, have multiple modes of invocation. The most familiar one is an interactive login shell. This is what you get when you ssh into a server, or open a TTY (Ctrl-Alt-F2 or so). Then, there’s what happens when you open a terminal while already logged in, which gets you an interactive shell. Then, then’s what happens when you run bash some-script.sh
, which is a non-interactive, non-login shell.
This is all very confusing, so here’s a table summarising when you will get each form of shell.
Interactive | Non-interactive | |
---|---|---|
Login | ssh , Linux TTY | bash -l script.sh |
Non-Login | Open a Terminal | bash script.sh ; qsub job.sh 1 |
Bash startup files
So far, so arcane. Where this matters, is that various programs need their initialisation code to be run before one can use them. The package managers conda
and mamba
are perfect examples, though there are others. Where this interacts with the above shell modes is that each mode will source a specific configuration file. The most famous of these is ~/.bashrc
, which is sourced for interactive, non-login shells. There is also ~/.bash_profile
, which is sourced for interactive login shells. See the bash manual for more info on these invocation modes and their respective startup files.
Again, this will make more sense with a table
Interactive | Non-interactive | |
---|---|---|
Login | ~/.bash_profile | ~/.bash_profile |
Non-Login | ~/.bashrc | ?????? |
The missing startup file?
You’ll notice the ????? in the bottom right square corresponds to running bash as a script, or e.g. via a job scheduler like SGE or PBS1. I say ??????
as there isn’t a file path that is loaded. Instead, what happens is that bash will source the file named in the environment variable $BASH_ENV
. What to do?!?
We can solve all this confusion by creating a file that I’ll name ~/.bash_env
, but you could name whatever you want. This file should contain all environmental variables you would like defined in any instance of bash. This will include things like export PATH=...
to include user-installed software (make sure this contains ~/.local/bin
), as well as other similar variables like MANPATH
, CPATH
, C_INCLUDE_PATH
, CPLUS_INCLUDE_PATH
, LD_LIBRARY_PATH
, and PKG_CONFIG_PATH
. You should also include any program initialisation code needed for programs that interact closely with your shell. These include things like conda
, mamba
, and lmod
(a.k.a module load
).
Then, in your ~/.bashrc
, we define export BASH_ENV=~/.bash_env
, and in ~/.bash_profile
we source ~/.bashrc
. This ensure that we get all the environmental variables, and all programs initialised, in every shell regardless of type. Note that zsh
actually formalises this, and has a hard-coded startup file to be sourced in any kind of shell named ~/.zsh_env
. Much more sensible…
The solution: BASH_ENV
Below, I post examples of these config files from a server I have an account on. You should be able to copy-paste these to your homedir on most linux systems, but be careful to make backups of any existing config files, and make sure you update the paths to match your server and copy any existing important configuration from your old config files. And obviously, the conda configuration below uses paths on our cluster, adjust to your system.
Another word of caution: many systems actually already define $BASH_ENV
, e.g. to enable a module system (like lmod) or something similar. In these cases, please take note of the value of $BASH_ENV
before doing this setup, and make sure your ~/.bash_env
sources these files (e.g. add source /usr/share/lmod/lmod/init/bash
to ~/.bash_env
, if $BASH_ENV
evaluated to /usr/share/lmod/lmod/init/bash
).
Reach out over email if you have any questions or something breaks.
Example ~/.bash_env
# BASH_ENV: for anything useful in *every* bash shell
# Update path env vars for each prefix of installed software.
INSTALL_PREFIXES=( "$HOME/.local" "/usr/lib/debian-med/")
for prefix in "${INSTALL_PREFIXES[@]}"
do
if [ -d "$prefix/bin" ] ; then
export PATH="$prefix/bin:$PATH"
fi
if [ -d "$prefix/lib" ] ; then
export LD_LIBRARY_PATH="$prefix/lib:$LD_LIBRARY_PATH"
fi
if [ -d "$prefix/include" ] ; then
export CPATH="$prefix/include:$CPATH"
export C_INCLUDE_PATH="$prefix/include:$C_INCLUDE_PATH"
export CPLUS_INCLUDE_PATH="$prefix/include:$CPLUS_INCLUDE_PATH"
fi
if [ -d "$prefix/lib/pkgconfig" ] ; then
export PKG_CONFIG_PATH="$prefix/lib/pkgconfig:$PKG_CONFIG_PATH" fi
if [ -d "$prefix/share/man" ] ; then
export MANPATH="$prefix/share/man:$MANPATH"
fi
done
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/usr/local/mambaforge-current/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
eval "$__conda_setup"
else
if [ -f "/usr/local/mambaforge-current/etc/profile.d/conda.sh" ]; then
. "/usr/local/mambaforge-current/etc/profile.d/conda.sh"
else
export PATH="/usr/local/mambaforge-current/bin:$PATH"
fi
fi
unset __conda_setup
# <<< conda initialize <<<
Example ~/.bashrc
# BASHRC: For things mainly useful for a human
#Aliases
alias ls="ls --color=auto"
alias l="ls -lhF"
alias ll="ls -lhFa"
alias gst="git status"
alias gaa="git annex add"
alias gas="git annex sync"
alias glol="git log --oneline --graph --decorate"
# use a two-line bash prompt
PS1="$PS1\n$ "
if [ -e "$HOME/.bash_env" ]
then
export BASH_ENV="$HOME/.bash_env"
source "$BASH_ENV"
fi
Example ~/.bash_profile
# Make any login shell behave the same as a typical interactive shell
if [ -e ~/.bashrc ] ; then source ~/.bashrc; fi