{ "cells": [ { "cell_type": "markdown", "id": "c29fe2a2-4704-4a79-8287-63e6711b62d4", "metadata": { "slideshow": { "slide_type": "slide" }, "tags": [] }, "source": [ "# introduction to the management of the execution environment with conda\n", "\n", "\n", "\"conda\"" ] }, { "cell_type": "markdown", "id": "c3e10fe3-e1a3-45fb-a245-164669bdbca8", "metadata": { "jp-MarkdownHeadingCollapsed": true, "slideshow": { "slide_type": "subslide" }, "tags": [] }, "source": [ "## Why using an environment manager?\n", "\n", "[Konrad Hinsen](https://10years.guix.gnu.org/video/guix-as-a-tool-for-computational-science/):\n", "\n", " “Code” is the code you care about. “Environment” is code you don’t care about. \n", "\n", "The problem is that all of **“the environment” influences the results** produced by the code you care about ...\n", "\n", "One solution: use an environment manager!
\n", "- avoid compilation and dependencies problems: an environment manager will take care of everything!
\n", "- have several environments in parallel, each with their own set of tools (or version)
\n", "- useful when cross-tools dependencies are incompatible with each other" ] }, { "cell_type": "markdown", "id": "6b9b9121-e956-4663-a4c1-4f028887ec30", "metadata": { "jp-MarkdownHeadingCollapsed": true, "slideshow": { "slide_type": "slide" }, "tags": [] }, "source": [ "## Conda definitions\n", "- **Environment**: a set of packages/tools in a directory (added to your PATH)
\n", "- **Conda**: an open source package + a general-purpose environment management system (installation, execution, upgrade). For any programming language, multi-platform (Windows, MacOS, Linux)
\n", "- **Conda package**: a compressed tarball of a tool
\n", "- **Conda channel**: the location where packages are stored" ] }, { "cell_type": "markdown", "id": "bb1d2b9f-b8df-43d2-bae9-4d57ff738b29", "metadata": { "jp-MarkdownHeadingCollapsed": true, "slideshow": { "slide_type": "subslide" }, "tags": [] }, "source": [ "## Conda access\n", "\n", "**Conda distributions**:
\n", "- Conda: environment management system, comes with a lot of packages
\n", "- Miniconda3: comes without installed packages (except conda)
\n", "- Mamba: a conda C-port (faster, more reproducible, & light-weighted version of conda)
\n", "- Micromamba: it is to mamba what miniconda is to conda
\n", "==> conda & mamba commands are transparent\n", "\n", "**How to download conda packages**
\n", "- [Anaconda/\"conda hub\"](https://anaconda.org/) (private company) a data science platform, stores conda packages of many domains (Machine Learning, Data Visualization, Dashboarding-web, Image Processing, Natural Language Processing, etc)
\n", "- made up of channels/owners. Each channel contains one or more conda packages
\n", "- be careful when downloading any package from an untrusted source, always inspect before installation" ] }, { "cell_type": "markdown", "id": "3ecdc4ca-0894-41ac-b6b2-012ec68b4207", "metadata": { "jp-MarkdownHeadingCollapsed": true, "slideshow": { "slide_type": "subslide" }, "tags": [] }, "source": [ "## About channels\n", "\n", "**Some conda channels:**
\n", "- **default** (not actually used, require a paid license)
\n", "- **conda-forge**: many popular conda packages (python but also R, perl, C, rust, ...)
\n", "- **bioconda**: bioinformaticians’ contributions
\n", "- **private**
\n", "\n", "**Channels list order:**
\n", "- when different channels have the same package ⇒ collisions
\n", "- collisions resolved following the order of your channels list + \"strict channel priority\" enabled" ] }, { "cell_type": "markdown", "id": "d79507b9-f752-4f6b-ac4d-10912b0bd4c4", "metadata": { "jp-MarkdownHeadingCollapsed": true, "slideshow": { "slide_type": "subslide" }, "tags": [] }, "source": [ "## Conda and R\n", "\n", "The R interpreter is included in the r-essentials packages (200 r-packages). Add ```r-``` before the regular R package name (eg. ```r-ggplot2```)\n", "\n", "## Conda limitations\n", "\n", "Conda enables the management (definition, creation, archiving) of software environments\n", "\n", "but ... is **still based on the OS**!\n", "\n", "This OS part is therefore missing to be fully reproducible but could be managed by a container solution.\n", "\n", "Other software environment managers, such as \"guix\", are also fully reproducible. \n", "\n", "\"pixi\n"," ] }, { "cell_type": "markdown", "id": "4e18db04-3ca9-4386-ba8e-85ad499d27b4", "metadata": { "jp-MarkdownHeadingCollapsed": true, "slideshow": { "slide_type": "slide" }, "tags": [] }, "source": [ "## Conda commands\n", "- initialisation of the shell: ```conda init bash```
\n", "- **creation** of a conda environment: ```conda create env -n myenv```
\n", "- **list environments** (* for the active one): ``` conda info --envs```
\n", "- **activate** the *myenv* environment: ```conda activate myenv```
\n", "- **list packages** (only in an active environment): ```conda list```
\n", "- **installation** of a tool/package: ```conda install package```
\n", "- **suppress a package** from the environment: ```conda remove package```
\n", "- **suppress the** *myenv* **environment**: ```conda env remove -n myenv```
\n", "- **inactivate** the environment: ```conda deactivate```
\n", "\n", "note: with the miniconda3 distribution environments are installed by default in a ```~/miniconda3/envs/``` repository" ] }, { "cell_type": "markdown", "id": "f5591284-d39d-43f3-b3ba-b67231b89b37", "metadata": { "slideshow": { "slide_type": "subslide" }, "tags": [] }, "source": [ "## 2 ways of using conda\n", "\n", "**interactive**
\n", "- create an environment
\n", "- activate the environment
\n", "- install some conda packages\n", "\n", "**configuration file**
\n", "- list all conda packages in a configuration file (```yml``` or ```json``` format)
\n", "- create the environment based on the configuration file (option ```-f```)
\n", "- activate the environment\n", "\n", "**reproducibility point of view:**
\n", "- use a configuration file (with the version of the package: ```::=```)
\n", "- set channel priority to _strict_
\n", "- save your conda creation environment files with your codes" ] }, { "cell_type": "markdown", "id": "f1202044-aa6f-489a-b48b-3a49ea3a11f5", "metadata": { "slideshow": { "slide_type": "slide" }, "tags": [] }, "source": [ "## a practical example\n", "\n", "You will:\n", "

1. if not yet done, initialize your conda shell
\n", "2. check that the tool is not present in your environment
\n", "3. create a config file to get the tool from conda
\n", "4. create a conda environment for the tool
\n", "5. activate the environment
\n", "6. use the tool (version)
\n", "7. quit
\n", "

\n", "\n", "For example choose the [multiqc](https://multiqc.info/) tool, which is often used in NGS analyses.\n", "\n", "Search the tool in the [\"conda hub\"/Anaconda platform](https://anaconda.org/conda-forge/hub), identify the channel & version.\n", "\n", "Next, you will use multiqc by the way of conda." ] }, { "cell_type": "markdown", "id": "558274a0-ebed-4f73-9f06-a8442352a7e7", "metadata": { "slideshow": { "slide_type": "subslide" }, "tags": [] }, "source": [ "## Conda access (practice)\n", "Conda is so used that it could even be installed by default to your machine. \n", "Try this: ```conda --version```\n", "\n", "Otherwise Conda is: \n", "- present in the `jupyter/minimal-notebook` docker container
\n", "- already activated on the IFB cluster, but to manage some environment variables, activate it (`module load conda`)
\n", "\n", "Before creating Conda environment, Conda need to know your shell version. But the ```conda init bash``` command need a close and re-open the terminal that doesn't work with the undelying terminal opened at the begining of this notebook.\n", "\n", "So for now, **open a term launcher** and copy/paste the notebook command line." ] }, { "cell_type": "markdown", "id": "395283d6-50b5-4465-9876-f0cdc0fa01bf", "metadata": { "slideshow": { "slide_type": "subslide" }, "tags": [] }, "source": [ "Before getting the tool, if you haven't already done so, initialize your shell for conda (choose bash):\n", "\n", "```conda init bash```" ] }, { "cell_type": "markdown", "id": "069e0acc-fc15-4a02-b8a0-199c0d7a0666", "metadata": { "slideshow": { "slide_type": "fragment" }, "tags": [] }, "source": [ "Check the absence of multiqc:" ] }, { "cell_type": "code", "execution_count": 4, "id": "2e24b1e5-f805-4493-b3e2-a2e5839c1cb8", "metadata": { "slideshow": { "slide_type": "fragment" }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "multiqc:" ] } ], "source": [ "%%sh\n", "cd ${PWD}\n", "whereis multiqc" ] }, { "cell_type": "markdown", "id": "c9cb93b0-9455-4d1c-9ec7-6fe25ba4a615", "metadata": { "slideshow": { "slide_type": "subslide" }, "tags": [] }, "source": [ "Using the channel & version you found on the conda hub, edit a yml file to guide the creation of the conda environment: \n", "\n", " cd ${PWD}\n", " echo \"name: env_multiqc_1.16\\nchannels:\\n - bioconda\\ndependencies:\\n - bioconda::multiqc=1.16\" > env_multiqc_1.16.yml\n", " more env_multiqc_1.16.yml" ] }, { "cell_type": "markdown", "id": "a347c20a-97b0-4b6f-bc0e-b5dec522b9cf", "metadata": { "slideshow": { "slide_type": "subslide" }, "tags": [] }, "source": [ "Manage the \"env_multiqc_1.16\" environment: 1) create 2) activate 3) use 4) quit:\n", "\n", " conda env list # list all conda env.\n", " conda env create -f env_multiqc_1.16.yml # create\n", " conda env list\n", " conda activate env_multiqc_1.16 # activate \n", " multiqc --help # use\n", " conda deactivate # quit\n", " multiqc --help # check the multiqc tool is not present" ] }, { "cell_type": "markdown", "id": "2f145531-583c-452e-a6bd-27e12ea13d87", "metadata": { "slideshow": { "slide_type": "slide" }, "tags": [] }, "source": [ "## Conclusion\n", "You've used conda to add a tool to your environment, and you've used it!\n", "\n", "By specifying a **configuration file** for the creation of the conda environment which contains the **version of the tool**, you have a FAIR approach (within the limitations of conda)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.4" } }, "nbformat": 4, "nbformat_minor": 5 }