Infrastructure
This page gives an overview of the conda-forge infrastructure, that is, an account of the various pieces maintained by the conda-forge contributors as well as third-party providers that collectively form the basis for the operation of conda-forge.
We start with the different Github repositories maintained by conda-forge itself, then describe the administrative commands available for use in those repositories, the so-called Admin web services, followed by the CI services, i.e. the third-party providers used for building and maintaining packages together. After that, we turn to a description of some aspects of the build environment for packages in Compilers and Runtimes, together with details about the upload to the package server.
Then, we see how the process of building a package interacts with different parts of the infrastructure.
We close out with a brief listing of involved entities.
Repositoriesโ
Staged-recipesโ
This repository is the gateway to conda-forge and where users can submit new recipes which, once reviewed and accepted, will generate a new feedstock and team. You can find the detailed guide for submitting new package recipes in The staging process.
- โ๏ธ Deployed in
conda-forge/staged-recipes
- ๐ Has access to Anaconda.org (cf-staging)
- ๐ค Integrated with
webservices
Anatomy of staged-recipesโ
recipes/
contains one or more subdirectories with user-submitted recipes.
Most cases will only submit one recipe at a time, but if several subdirectories are present, the build_all.py
script will build them in the right order so dependencies are satisfied.
.ci_support
contains the conda-build YAML configuration files, but in this case (if compared to feedstocks), you will also find some scripts:
build_all.py
: Calls conda-build in the right (topographically sorted) order.compute_build_graph.py
: Supportsbuild_all.py
by providing the job graph with all the submitted recipes.
The YAML files included in .ci_support
are minimal and not rendered like the ones you find in feedstocks.
Instead, conda-build will take these and combine them with the pinnings from conda-forge-pinning
at runtime.
Also note that staged-recipes
only builds for x64. Support for additional architectures can only be done once a feedstock has been provided.
- Linux:
linux64.yaml
plus the CUDA (10.2, 11.0, 11.1 and 11.2) variants. - macOS:
osx64.yaml
. - Windows
win64.yaml
.
The directory .scripts
contains roughly the same shell scripts that would be used in a feedstock for the CI pipelines.
However, since staged-recipes
does not support rerendering, these are kept in sync manually and it is common to see some differences.
Workflowsโ
The main job run on staged-recipes
is the conda-build
job that runs on every PR (and push to main
) to check whether the recipes build packages correctly.
These jobs run on Azure Pipelines defined in .azure-pipelines/
.
The actual creation of the feedstock is run in conda-forge/admin-requests.
Additional workflows help users set up their recipes correctly. They react to events in PRs:
automate-review-labels
: Updates PR labels to streamline reviews and requests for help.correct_directory
: Posts a PR comment ifmeta.yaml
and friends were not added in arecipes/
subdirectory.do_not_edit_example
: Posts a PR comment if therecipes/example/
recipe was edited.
External services connect to staged-recipes
too:
- The
@conda-forge-admin
bot (deployed atwebservices
) will lint and provide hints in PRs based on the contents of the recipe.
Feedstocksโ
- โ๏ธ Deployed in Github repositories
- ๐ Has access to Azure Pipelines, Github Actions, Anaconda.org (cf-staging)
- ๐ Might have access to Travis CI, Cirun via
admin-requests
(WIP) - ๐ค Integrated with
admin-migrations
,admin-requests
, theautotick-bot
, andwebservices
.
Conda-forge has thousands of feedstocks. Each feedstock hosts a recipe plus the required pipelines, supporting scripts and configuration metadata.
The contents of a feedstock are well specified. Only two locations are user-managed:
recipe/
: Contains the conda-build instructions to build packages. It needs, at least, ameta.yaml
file, and this is also where the optionalconda_build_config.yaml
usually goes.conda-forge.yml
: This is the feedstock configuration file.
You should never manually edit files not listed above! Changes will be overridden in the next feedstock rerender.
Combining these two sources with some external components, conda-smithy
will generate (render) the contents of the feedstock. Many of the directories are named like that because it is what the external service (e.g. Azure) requests. However, some conda-smithy
-unique directories are worth discussing:
.ci_support/
: Contains the renderedconda_build_config.yaml
files, passed toconda-build
via the-m
flag. Each file here corresponds to one job in the CI build matrix..ci_support/migrations/
: Special YAML files that instructconda-smithy
how to update the.ci_support/*.yaml
files. These migration files are usually put here by theautotick-bot
infrastructure, and removed once the migration is considered finished..scripts/
: Common logic and code supporting the steps you can find in the CI pipelines and local debugging tools.build-locally.py
: A Python script to debug recipes in your machine, roughly equivalent to what's done in the CI pipelines.
- Rerendering a feedstock
- Recommended workflow
feedstocks monorepoโ
A single repository containing all feedstocks as submodules.
- โ๏ธ Deployed in
conda-forge/feedstocks
feedstock-outputsโ
This repository is a registry of feedstock names and the packages (artifacts) they produce.
- โ๏ธ Deployed in Github Actions via
conda-forge/feedstock-outputs
- ๐ Has access to Azure, Anaconda.org (cf-staging)
Its main purpose is to provide an allow-list for the validation server to prevent malicious cross-feedstock builds, although it's also an informative map of feedstocks <-> packages
that is exposed in the packages section of the website.
cdt-buildsโ
- โ๏ธ Deployed in Azure Pipelines via
conda-forge/cdt-builds
- ๐ Has access to Azure Pipelines, Anaconda.org (cf-staging)
This special repository builds Core Dependency Tree packages for conda-forge (Linux only). It doesn't use the feedstock automated machinery. Instead, it has its own Azure Pipelines workflow and a well-documented README.
msys2-recipesโ
- โ๏ธ Deployed manually from
conda-forge/msys2-recipes
This is a fork of the old community recipes repository at Anaconda, which includes the msys2
recipes under the msys2/
directory.
Note also the supporting scripts in the common-scripts/
folder.
Websiteโ
The current conda-forge.org is a statically generated website published to Github Pages.
- ๐ Source at conda-forge/conda-forge.github.io
- โ๏ธ Deployed in conda-forge.org
- ๐ค to enhance the utility of the documentation we also use
- PR previews at Netlify
- Statistics at GoatCounter
- Search powered by Algolia
The documentation is built with Docusaurus and the source files are located in the docs/
directory of the repository.
If you find any typos, errors, unclear explanations, or new topics that can be covered, you can suggest changes to the documentation. For more details, please refer to Improve the documentation.
In addition to the static documentation, the website also offers information on the current status of conda-forge as well as a mapping of packages to feedstocks.
- Status: conda-forge.org/status
- Packages-to-feedstock mapping: conda-forge.org/feedstock-outputs
Metadata repositoriesโ
These are repositories that primarily hold metadata used by other parts of the conda-forge ecosystem.
conda-forge pinningโ
Hosts the global pinnings for conda-forge, and the ongoing migrations.
- โ๏ธ Deployed in Anaconda.org via
conda-forge/conda-forge-pinning-feedstock
- ๐ Has access to Azure, Anaconda.org (cf-staging)
Package-wide dependency pins are defined in conda_build_config.yaml in the conda-forge/conda-forge-pinning-feedstock.
For more information on conda-forge wide package pins, please refer to Globally pinned packages.
Please open a PR and/or an issue there, if you think a pin needs to be advanced. For more information on updating globally pinned packages, please refer to Updating package pins.
conda-forge-repodata-patchesโ
This repository creates the repodata.json
patches used by the Anaconda.org to amend the metadata coming from the published packages.
- โ๏ธ Deployed in Anaconda.org via
conda-forge/conda-forge-repodata-patches-feedstock
- ๐ Has access to Azure, Anaconda.org (cf-staging)
conda-forge-ci-setupโ
This special feedstock provides a package that defines the logic to install and configure a common CI setup across providers.
- โ๏ธ Deployed in Anaconda.org via
conda-forge/conda-forge-ci-setup-feedstock
- ๐ Has access to Azure, Anaconda.org (cf-staging)
regro/cf-graph-countyfairโ
This is the graph data used by autotick-bot
.
- โ๏ธ Deployed in Github Actions via
regro/cf-graph-countyfair
- โ Needs
regro/cf-scripts
,conda-forge/conda-forge-pinning-feedstock
- ๐ค Uses
@regro-cf-autotick-bot
- ๐ Has access to Github API
The logic to build the graph is provided by cf-scripts
.
docker-imagesโ
This repository builds the Docker images used to provide a unified system on all Linux builds.
- โ๏ธ Deployed in
conda-forge/docker-images
- ๐ Has access to DockerHub and Quay.io
- โ Needed by
staged-recipes
, feedstocks.
Code repositoriesโ
These are repositories that hold programs and other codes that define behavior. However, their actions are often not triggered here, but rather used by other parts of the conda-forge ecosystem.
Smithyโ
This is the main feedstock creation and maintenance tool.
- ๐ Source at
conda-forge/conda-smithy
- ๐ฆ Packaged at
conda-forge/conda-smithy-feedstock
- ๐ Documentation
- ๐ conda-forge.yml Documentation
Most of its usage is automated by our infrastructure:
- Feedstock creation and services registration at
staged-recipes
- Regeneration (rerendering), linting and hinting in PRs done by
conda-forge-admin
onwebservices
However, you can also use it locally or on your forge-like deployments. For local debugging, you will find these commands useful:
conda-smithy rerender
conda-smithy recipe-lint
Smithy contains maintenance code for conda-forge, which is used by the conda-smithy
command line tool and the Admin web services.
conda-forge/conda-smithy
is the right repository to report bugs for
- The rerendering process
- The recipe linter
- CI support utils
conda-smithy
also contains the command line tool that you should use if you rerender manually from the command line (see Rerendering feedstocks).
Smithy can be used beyond conda-forge's purposes. For example, it can be used to set up self-hosted Azure agents for non-conda-forge infrastructures. (You could also consider using Azure virtual machine scale set agents, which could be less expensive to run than permanently active agents.)
Web servicesโ
The Heroku app providing the conda-forge web services lives in conda-forge/conda-forge-webservices.
Please note that the code logic provided by the app is in the Smithy
repository.
Bugs or suggestions regarding the service functionality should therefore be opened in conda-forge/conda-smithy
's bug tracker.
regro/cf-scriptsโ
The code and logic behind autotick-bot
.
- ๐ Source at
regro/cf-scripts
- ๐ Documentation
Automated maintenanceโ
These components perform actions in automated ways, either triggered by a specific event or continuously as part of a loop.
admin-migrationsโ
- โ๏ธ Deployed in Github Actions via
conda-forge/admin-migrations
- ๐ค Uses
@conda-forge-curator
- ๐ Has access to Github API, Anaconda.org (conda-forge and cf-staging), Circle, Travis, Azure
This repository hosts workflows that are running 24/7. Its job is to procure an automation loop where some maintenance tasks are added. Its main user is the core team.
admin-requestsโ
- โ๏ธ Deployed in Github Actions via
conda-forge/admin-requests
- ๐ค Uses
@conda-forge-curator
- ๐ Has access to Github API, Anaconda.org (conda-forge and cf-staging), Circle, Travis, Azure
This repository hosts workflows that mainly run when triggered by a user-initiated action. This is usually done via a PR that, once approved, is merged and triggers the requested action (mark a package as broken, archive a feedstock, etc).
It also does the job of creating new feedstocks for recipes that have been merged in conda-forge/staged-recipes
.
The create_feedstocks
workflow runs several times per hour to create the new feedstock repositories on the conda-forge
organization.
The core logic is defined in the Python script .github/workflows/scripts/create_feedstocks.py
.
autotick-botโ
- โ๏ธ Deployed in
regro/cf-scripts
- โ Needs
regro/cf-scripts
,regro/cf-graph-countyfair
,conda-forge/conda-forge-pinning-feedstock
- ๐ค Uses
@regro-cf-autotick-bot
The older repo regro/autotick-bot
is no longer in use;
the bot now runs directly in regro/cf-scripts
.
webservicesโ
- โ๏ธ Deployed in Heroku Dyno (
conda-forge.herokuapp.com
) - โ Needs
conda-forge/conda-forge-webservices
- ๐ค Uses
@conda-forge-webservices
,@conda-forge-admin
- ๐ Has access to Github API, Anaconda.org (cf-staging and conda-forge), Heroku
This web application powers several services, like:
- the
@conda-forge-admin
bot and its@conda-forge-admin, please ...
commands - the
cf-staging
toconda-forge
validation (plus copy) - status monitoring
Admin web servicesโ
conda-forge is running a webservice on Heroku called conda-forge-webservices.
The following services are run by default on a feedstock:
- It will lint the recipes in the PRs and report back whether the recipe is in excellent condition or not.
- When maintainers are added to a recipe, each of the maintainers will be added to the team and given push access to the feedstock.
The webservice also listens to issues and PR comments, so that you can ask for the following services to be done:
@conda-forge-admin, please rerenderโ
Entering the above phrase in a PR of a feedstock will rerender the feedstock and push the changes to your PR.
Make sure to tick the Allow edits from maintainers
button located at the bottom of the right side bar of
the PR. If you enter this phrase in the comment for an issue, the bot will create a new pull request, with the requested
re-rendering being completed.
@conda-forge-admin, please add noarch: pythonโ
Entering the above phrase in a PR or an issue of a feedstock will add noarch: python
to the build and rerender the feedstock
for you.
@conda-forge-admin, please lintโ
Entering the above phrase in a PR of a feedstock will lint the PR again.
@conda-forge-admin, please update teamโ
Entering the above phrase in an issue will update the team for the feedstock. This is usually done automatically.