+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ This section goes over the process of composing a release.
+ +This page goes over empanadas, which is part of the SIG/Core toolkit. Empanadas
+assists SIG/Core is composing repositories, creating ISO's, creating images, and
+various other activities in Rocky Linux. It is also used for general testing and
+debugging of repositories and its metadata.
| + | + |
|---|---|
| Owner | +SIG/Core (Release Engineering & Infrastructure) | +
| Email Contact | +releng@rockylinux.org | +
| Mattermost Contacts | +@label @neil |
+
| Mattermost Channels | +~Development |
+
empanadas is a python project using poetry, containing various built-in modules
+with the goal to try to emulate the Fedora Project's pungi to an extent. While it
+is not perfect, it achieves the very basic goals of creating repositories, images
+and ISO's for consumption by the end user. It also has interactions with peridot,
+the build system used by the RESF to build the Rocky Linux distribution.
For performing syncs, it relies on the use of podman to perform syncing in a +parallel fashion. This was done because it is not possible to run multiple dnf +transactions at once on a single system and looping one repository at a time is +not sustainable (nor fast).
+fpart must be installed on the system (available in EPEL on EL systems)mock must be installed if building live images%rhel macro setAs of this writing, empanadas has the following abilities:
loraxxorriso for DVD and minimal imageslivemedia-creator and mockThe below is how to install empanadas from the development branch on a Fedora +system.
+% dnf install git podman fpart poetry mock -y
+% git clone https://git.resf.org/sig_core/toolkit.git -b devel
+% cd toolkit/iso/empanadas
+% poetry install
+Depending on how you are using empanadas will depend on how your configurations +will be setup.
+empanadas/common.pyempanadas/config/*.yamlempanadas/sig/*.yamlThese configuration files are delicate and can control a wide variety of the +moving parts of empanadas. As these configurations are fairly massive, we +recommend checking the reference guides for deeper details into +configuring for base distribution or "SIG" content.
+The most common way to use empanadas is to sync repositories from a peridot
+instance. This is performed upon each release or on each set of updates as they
+come from upstream. Below lists how to use empanadas, as well as the common
+options.
Note that for each of these commands, it is fully expected you are running
+poetry run in the root of empanadas.
URL: https://accounts.rockylinux.org
+Purpose: Account Services maintains the accounts for almost all components of the Rocky ecosystem
+Technology: Noggin used by Fedora Infrastructure
+Contact: ~Infrastructure in Mattermost and #rockylinux-infra in Libera IRC
URL: https://git.resf.org
+Purpose: General projects, code, and so on for the Rocky Enterprise Software Foundation.
+Technology: Gitea
+Contact: ~Infrastructure, ~Development in Mattermost and #rockylinux-infra, #rockylinux-devel in Libera IRC
URL: https://github.com/rocky-linux
+Purpose: General purpose code, assets, and so on for Rocky Linux. Some content is mirrored to the RESF Git Service.
+Technology: GitHub
+Contact: ~Infrastructure, ~Development in Mattermost and #rockylinux-infra, #rockylinux-devel in Libera IRC
URL: https://git.rockylinux.org
+Purpose: Packages and light code for the Rocky Linux distribution
+Technology: GitLab
+Contact: ~Infrastructure, ~Development in Mattermost and #rockylinux-infra, #rockylinux-devel in Libera IRC
Purpose: Users can subscribe and interact with various mail lists for the Rocky ecosystem
+Technology: Mailman 3 + Hyper Kitty
+Contact: ~Infrastructure in Mattermost and #rockylinux-infra in Libera IRC
| Name | +Mattermost Name | +IRC Name | +|
|---|---|---|---|
| Louis Abel | +label@rockylinux.org | +@nazunalika | +Sokel/label/Sombra | +
| Mustafa Gezen | +mustafa@rockylinux.org | +@mustafa | +mstg | +
| Skip Grube | +skip@rockylinux.org | +@skip77 | ++ |
| Sherif Nagy | +sherif@rockylinux.org | +@sherif | ++ |
| Pablo Greco | +pgreco@rockylinux.org | +@pgreco | +pgreco | +
| Neil Hanlon | +neil@resf.org | +@neil | +neil | +
| Taylor Goodwill | +tg@resf.org | +@tgo | +tg | +
This section goes over at a high level how we compose releases for Rocky Linux. +As most of our tools are home grown, we have made sure that the tools are open +source and in our git services.
+This page should serve as an idea of the steps we generally take and we hope +that other projects out there who wish to also use our tools can make sure they +can use them in this same way, whether they want to be an Enterprise Linux +derivative or another project entirely.
+The tools in use for the distribution are in the table below.
+| Tool | +Maintainer | +Code Location | +
|---|---|---|
| srpmproc | +SIG/Core at RESF | +GitHub | +
| empanadas | +SIG/Core at RESF | +sig-core-toolkit | +
| Peridot | +SIG/Core at RESF | +GitHub | +
| MirrorManager 2 | +Fedora Project | +MirrorManager2 | +
For Rocky Linux to be build, we use Peridot as the build system and
+empanadas to "compose" the distribution. As we do not use Koji for Rocky
+Linux beyond version 9, pungi can no longer be used. Peridot instead takes
+pungi configuration data and comps and transforms them into a format it
+can understand. Empanadas then comes in to do the "compose" and sync
+all the repositories down.
Step by step, it looks like this:
+In Parallel:
+Lorax results are pulled down from an S3 bucket
+Step by step, it looks like this:
+URL: https://accounts.rockylinux.org
+Purpose: Account Services maintains the accounts for almost all components of the Rocky ecosystem
+Technology: Noggin used by Fedora Infrastructure
+Contact: ~Infrastructure in Mattermost and #rockylinux-infra in Libera IRC
URL: https://git.resf.org
+Purpose: General projects, code, and so on for the Rocky Enterprise Software Foundation.
+Technology: Gitea
+Contact: ~Infrastructure, ~Development in Mattermost and #rockylinux-infra, #rockylinux-devel in Libera IRC
URL: https://github.com/rocky-linux
+Purpose: General purpose code, assets, and so on for Rocky Linux. Some content is mirrored to the RESF Git Service.
+Technology: GitHub
+Contact: ~Infrastructure, ~Development in Mattermost and #rockylinux-infra, #rockylinux-devel in Libera IRC
URL: https://git.rockylinux.org
+Purpose: Packages and light code for the Rocky Linux distribution
+Technology: GitLab
+Contact: ~Infrastructure, ~Development in Mattermost and #rockylinux-infra, #rockylinux-devel in Libera IRC
Purpose: Users can subscribe and interact with various mail lists for the Rocky ecosystem
+Technology: Mailman 3 + Hyper Kitty
+Contact: ~Infrastructure in Mattermost and #rockylinux-infra in Libera IRC
| Name | +Mattermost Name | +IRC Name | +|
|---|---|---|---|
| Louis Abel | +label@rockylinux.org | +@nazunalika | +Sokel/label/Sombra | +
| Mustafa Gezen | +mustafa@rockylinux.org | +@mustafa | +mstg | +
| Skip Grube | +skip@rockylinux.org | +@skip77 | ++ |
| Sherif Nagy | +sherif@rockylinux.org | +@sherif | ++ |
| Pablo Greco | +pgreco@rockylinux.org | +@pgreco | +pgreco | +
| Neil Hanlon | +neil@resf.org | +@neil | +neil | +
| Taylor Goodwill | +tg@resf.org | +@tgo | +tg | +
This page goes over the Peridot Build System and how SIG/Core utilizes it.
+More to come.
+ +In some cases, a package has to be rebuilt. A package may be rebuilt for these reasons:
+This typically applies to packages being built from a given src subgroup. Packages pulled from upstream don't fall into this category in normal circumstances. In those cases, they receive .0.1 and so on as standalone rebuilds.
The common.py configuration contains dictionaries and classes that dictate
+most of the functionality of empanadas.
type: Dictionary
+type: String
+required: True
+description: Empanadas expects to run on an EL system. This is part of the +general check up. It should not be hardcoded and use the rpm python module.
+type: String
+required: False
+description: Was the original tag placed in mock configs. This combines el
+with the rpm python module expansion. This is no longer required. The
+option is still available for future use.
type: String
+required: True
+description: The architecture of the current running system. This is checked +against the supported architectures in general release configurations. This +should not be hardcoded.
+type: String
+required: True
+description: Date time stamp in the form of YYYYMMDD.HHMMSS. This should not be +hardcoded.
+type: String
+required: True
+description: Root path of composes on the system running empanadas.
+type: String
+required: False
+description: For future use. Root path of staging repository location where +content will be synced to.
+type: String
+required: False
+description: For future use. Root path of production repository location where +content will be synced to from staging.
+type: String
+required: True
+description: For future use. Stub path that is appended to staging_root and
+production_root.
example: mirror/pub/rocky
type: String
+required: True
+description: For future use. Stub path that is appended to staging_root and
+production_root for SIG content.
example: mirror/pub/sig
type: String
+required: True
+description: URL to the base url's where the repositories live. This is
+typically to a peridot instance. This is supplemented by the configuration
+project_id parameter.
Note that this does not have to be a peridot instance. The combination of this
+value and project_id can be sufficient enough for empanadas to perform its
+work.
type: String
+required: True
+description: Hardcoded path to where ISO work is performed within a mock chroot. +This is the default path created by mock and it is recommended not to change +this.
+example: /builddir
type: String
+required: True
+description: This is the container used to perform all operations in podman.
+example: centos:stream9
type: String
+required: True
+description: Name of the distribution you are building or building for.
+example: Rocky Linux
type: String
+required: True
+description: Short name of the distribution you are building or building for.
+example: Rocky
type: Dictionary
+required: True
+description: Translates Linux architectures to golang architectures. Reserved +for future use.
+type: String
+required: False
+description: Region you are working in with AWS or onprem cloud that supports +this variable.
+example: us-east-2
type: String
+required: False
+description: Name of the S3-compatible bucket that is used to pull images from.
+Requires aws_region.
type: String
+required: False
+description: URL of the S3-compatible bucket that is used to pull images from.
+type: Dictionary
+description: Key value pairs of cloud or image variants. The value is either
+None or a list type.
config = {
+ "rlmacro": rpm.expandMacro('%rhel'),
+ "dist": 'el' + rpm.expandMacro('%rhel'),
+ "arch": platform.machine(),
+ "date_stamp": time.strftime("%Y%m%d.%H%M%S", time.localtime()),
+ "compose_root": "/mnt/compose",
+ "staging_root": "/mnt/repos-staging",
+ "production_root": "/mnt/repos-production",
+ "category_stub": "mirror/pub/rocky",
+ "sig_category_stub": "mirror/pub/sig",
+ "repo_base_url": "https://yumrepofs.build.resf.org/v1/projects",
+ "mock_work_root": "/builddir",
+ "container": "centos:stream9",
+ "distname": "Rocky Linux",
+ "shortname": "Rocky",
+ "translators": {
+ "x86_64": "amd64",
+ "aarch64": "arm64",
+ "ppc64le": "ppc64le",
+ "s390x": "s390x",
+ "i686": "386"
+ },
+ "aws_region": "us-east-2",
+ "bucket": "resf-empanadas",
+ "bucket_url": "https://resf-empanadas.s3.us-east-2.amazonaws.com"
+}
+
+ALLOWED_TYPE_VARIANTS = {
+ "Azure": None,
+ "Container": ["Base", "Minimal", "UBI"],
+ "EC2": None,
+ "GenericCloud": None,
+ "Vagrant": ["Libvirt", "Vbox"],
+ "OCP": None
+
+}
+Each file in empanads/config/ is a yaml file that contains configuration
+items for the distribution release version. The configuration can heavily
+dictate the functionality and what features are directly supported by empanadas
+when ran.
See the items below to see which options are mandatory and optional.
+The Top Level is the name of the profile and starts the YAML dictionary for the +release. It is alphanumeric and accepts punctuation within reason. Common +examples:
+99-beta8-lookaheadtype: String
+required: True
+description: Needed for treeinfo and discinfo generation.
+type: String
+required: True
+description: Full version of a release
+type: String
+required: True
+description: Release Candidate or Beta descriptor. Sets names and versions with +this descriptor if enabled.
+type: String
+required: True
+description: Major version of a release
+type: String
+required: True
+description: Minor version of a release
+type: String
+required: True
+description: Matches the top level of the release. This should not differ from +the top level assignment.
+type: String
+required: True
+description: Sets the dist tag for mock configs.
+type: String
+required: True
+description: A URL to the bug tracker for this release or distribution.
+type: String
+required: True
+description: Checksum type. Used when generating checksum information for +images.
+type: String
+required: False
+description: For future use with icicle.
+type: list
+required: True
+description: List of supported architectures for this release.
+type: boolean
+required: True
+description: Sets if architecture x86_64 will be multilib. It is recommended
+that this is set to True.
type: String
+required: True
+description: Appended to the base repo URL in common.py. For peridot, it is the +project id that is generated for the project you are pulling from. It can be set +to anything else if need be for non-peridot use.
+type: dict
+required: False
+description: For future use. Sets symlinks to repositories for backwards +compatibility. Key value pairs only.
+type: dict
+required: False
+description: Renames a repository to the value set. For example, renaming all
+to devel. Set to {} if no renames are goign to occur.
type: list
+required: True
+description: List of repositories that will be synced/managed by empanadas.
+type: dict
+required: True
+description: Key value pairs of packages and repodata. These are appended
+appropriately during syncing and ISO actions. Setting these are mandatory.
type: dictionary
+required: True if building ISO's and operating with lorax.
+description: Controls how lorax and extra ISO's are built.
+If are you not building images, set to {}
type: boolean
+required: True
+description: Dictates of xorrisofs is used to build images. Setting to false +uses genisoimage. It is recommended that xorrisofs is used.
+type: boolean
+required: True
+description: Set to false if you are using xorrisofs. Can be set to true when +using genisoimage.
+type: dict
+required: True
+description: Dictates the ISO images that will be made or the treeinfo that will +be generated.
+Note: The primary repository (for example, BaseOS) will need to be listed to
+ensure the treeinfo data is correctly generated. disc should be set to False
+and isoskip should be set to True. See the example section for an example.
type: boolean
+required: True
+description: This tells the iso builder if this will be a generated ISO.
+type: boolean
+required: False
+description: This tells the iso builder if this will be skipped, even if disc
+is set to True. Default is False.
type: string
+required: True
+description: Names the primary variant repository for the image. This is set in +.treeinfo.
+type: list
+required: True
+description: Names of the repositories included in the image. This is added to +.treeinfo.
+type: string
+required: True
+required value: dvd
description: This is required if building more than the DVD image. By default,
+the the name dvd is harcoded in the buildImage template.
type: dict
+required: True if building lorax images.
+description: Sets up lorax images and which repositories to use when building +lorax images.
+type: list
+required: True
+description: List of repos that are used to pull packages to build the lorax +images.
+type: string
+required: True
+description: Base repository for the release
+type: list
+required: False
+description: Excludes packages that are not needed when lorax is running.
+type: list
+required: True
+description: Required list of installed packages needed to build lorax images.
+type: dict
+required: False
+description: Dictates what live images are built and how they are built.
+type: string
+required: True
+description: The git repository URL where the kickstarts live
+type: string
+required: True
+description: The branch being used for the kickstarts
+type: dict
+required: True
+description: Key value pairs of the live images being created. Key being the +name of the live image, value being the kickstart name/path.
+type: list
+required: True
+description: List of allowed architectures that will build for the live images.
+type: list
+required: True
+description: Required list of packages needed to build the live images.
+type: dict
+required: False
+description: Cloud related settings.
+Set to {} if not needed.
type: dict
+required: True
+description: Cloud images that will be generated and in a bucket to be pulled, +and their format.
+type: dict
+required: True
+description: Name of the cloud image being pulled.
+Accepted key value options:
+format, which is raw, qcow2, vhd, tar.xzvariants, which is a listprimary_variant, which symlinks to the "primary" variant in the variant listtype: dict
+required: True
+description: Repoclosure settings. These settings are absolutely required when +doing full syncs and need to check repositories for consistency.
+type: dict
+required: True
+description: For each architecture (key), dnf switches/settings that dictate how +repoclosure will check for consistency (value, string).
+example: x86_64: '--forcearch=x86_64 --arch=x86_64 --arch=athlon --arch=i686 --arch=i586 --arch=i486 --arch=i386 --arch=noarch'
type: dict
+required: True
+description: For each repository that is pulled for a given release(key),
+repositories that will be included in the repoclosure check. A repository that
+only checks against itself must have a value of [].
type: dict
+required: True
+description: Extra files settings and where they come from. Git repositories are +the only supported method.
+type: string
+required: True
+description: URL to the git repository with the extra files.
+type: string
+required: True
+description: URL to the git repository with the extra files, but the "raw" url +form.
+example: git_raw_path: 'https://git.rockylinux.org/staging/src/rocky-release/-/raw/r9/'
type: string
+required: True
+description: Branch where the extra files are pulled from.
+type: dict
+required: True
+description: For each gpg key type (key), the relative path to the key in the +git repository (value).
+These keys help set up the repository configuration when doing syncs.
+By default, the RepoSync class sets stable as the gpgkey that is used.
type: list
+required: True
+description: List of files from the git repository that will be used as "extra" +files and placed in the repositories and available to mirrors and will appear on +ISO images if applicable.
+---
+'9':
+ fullname: 'Rocky Linux 9.0'
+ revision: '9.0'
+ rclvl: 'RC2'
+ major: '9'
+ minor: '0'
+ profile: '9'
+ disttag: 'el9'
+ bugurl: 'https://bugs.rockylinux.org'
+ checksum: 'sha256'
+ fedora_major: '20'
+ allowed_arches:
+ - x86_64
+ - aarch64
+ - ppc64le
+ - s390x
+ provide_multilib: True
+ project_id: '55b17281-bc54-4929-8aca-a8a11d628738'
+ repo_symlinks:
+ NFV: 'nfv'
+ renames:
+ all: 'devel'
+ all_repos:
+ - 'all'
+ - 'BaseOS'
+ - 'AppStream'
+ - 'CRB'
+ - 'HighAvailability'
+ - 'ResilientStorage'
+ - 'RT'
+ - 'NFV'
+ - 'SAP'
+ - 'SAPHANA'
+ - 'extras'
+ - 'plus'
+ structure:
+ packages: 'os/Packages'
+ repodata: 'os/repodata'
+ iso_map:
+ xorrisofs: True
+ iso_level: False
+ images:
+ dvd:
+ disc: True
+ variant: 'AppStream'
+ repos:
+ - 'BaseOS'
+ - 'AppStream'
+ minimal:
+ disc: True
+ isoskip: True
+ repos:
+ - 'minimal'
+ - 'BaseOS'
+ variant: 'minimal'
+ volname: 'dvd'
+ BaseOS:
+ disc: False
+ isoskip: True
+ variant: 'BaseOS'
+ repos:
+ - 'BaseOS'
+ - 'AppStream'
+ lorax:
+ repos:
+ - 'BaseOS'
+ - 'AppStream'
+ variant: 'BaseOS'
+ lorax_removes:
+ - 'libreport-rhel-anaconda-bugzilla'
+ required_pkgs:
+ - 'lorax'
+ - 'genisoimage'
+ - 'isomd5sum'
+ - 'lorax-templates-rhel'
+ - 'lorax-templates-generic'
+ - 'xorriso'
+ cloudimages:
+ images:
+ EC2:
+ format: raw
+ GenericCloud:
+ format: qcow2
+ livemap:
+ git_repo: 'https://git.resf.org/sig_core/kickstarts.git'
+ branch: 'r9'
+ ksentry:
+ Workstation: rocky-live-workstation.ks
+ Workstation-Lite: rocky-live-workstation-lite.ks
+ XFCE: rocky-live-xfce.ks
+ KDE: rocky-live-kde.ks
+ MATE: rocky-live-mate.ks
+ allowed_arches:
+ - x86_64
+ - aarch64
+ required_pkgs:
+ - 'lorax-lmc-novirt'
+ - 'vim-minimal'
+ - 'pykickstart'
+ - 'git'
+ variantmap:
+ git_repo: 'https://git.rockylinux.org/rocky/pungi-rocky.git'
+ branch: 'r9'
+ git_raw_path: 'https://git.rockylinux.org/rocky/pungi-rocky/-/raw/r9/'
+ repoclosure_map:
+ arches:
+ x86_64: '--forcearch=x86_64 --arch=x86_64 --arch=athlon --arch=i686 --arch=i586 --arch=i486 --arch=i386 --arch=noarch'
+ aarch64: '--forcearch=aarch64 --arch=aarch64 --arch=noarch'
+ ppc64le: '--forcearch=ppc64le --arch=ppc64le --arch=noarch'
+ s390x: '--forcearch=s390x --arch=s390x --arch=noarch'
+ repos:
+ devel: []
+ BaseOS: []
+ AppStream:
+ - BaseOS
+ CRB:
+ - BaseOS
+ - AppStream
+ HighAvailability:
+ - BaseOS
+ - AppStream
+ ResilientStorage:
+ - BaseOS
+ - AppStream
+ RT:
+ - BaseOS
+ - AppStream
+ NFV:
+ - BaseOS
+ - AppStream
+ SAP:
+ - BaseOS
+ - AppStream
+ - HighAvailability
+ SAPHANA:
+ - BaseOS
+ - AppStream
+ - HighAvailability
+ extra_files:
+ git_repo: 'https://git.rockylinux.org/staging/src/rocky-release.git'
+ git_raw_path: 'https://git.rockylinux.org/staging/src/rocky-release/-/raw/r9/'
+ branch: 'r9'
+ gpg:
+ stable: 'SOURCES/RPM-GPG-KEY-Rocky-9'
+ testing: 'SOURCES/RPM-GPG-KEY-Rocky-9-Testing'
+ list:
+ - 'SOURCES/Contributors'
+ - 'SOURCES/COMMUNITY-CHARTER'
+ - 'SOURCES/EULA'
+ - 'SOURCES/LICENSE'
+ - 'SOURCES/RPM-GPG-KEY-Rocky-9'
+ - 'SOURCES/RPM-GPG-KEY-Rocky-9-Testing'
+...
+Each file in empanads/sig/ is a yaml file that contains configuration
+items for the distribution release version. The configuration determines the
+structure of the SIG repositories synced from Peridot or a given repo.
Note that a release profile (for a major version) is still required for this +sync to work.
+See the items below to see which options are mandatory and optional.
+Use this section to locate reference configuration items for the toolkit.
+ +URL: https://accounts.rockylinux.org
+Purpose: Account Services maintains the accounts for almost all components of the Rocky ecosystem
+Technology: Noggin used by Fedora Infrastructure
+Contact: ~Infrastructure in Mattermost and #rockylinux-infra in Libera IRC
URL: https://git.resf.org
+Purpose: General projects, code, and so on for the Rocky Enterprise Software Foundation.
+Technology: Gitea
+Contact: ~Infrastructure, ~Development in Mattermost and #rockylinux-infra, #rockylinux-devel in Libera IRC
URL: https://github.com/rocky-linux
+Purpose: General purpose code, assets, and so on for Rocky Linux. Some content is mirrored to the RESF Git Service.
+Technology: GitHub
+Contact: ~Infrastructure, ~Development in Mattermost and #rockylinux-infra, #rockylinux-devel in Libera IRC
URL: https://git.rockylinux.org
+Purpose: Packages and light code for the Rocky Linux distribution
+Technology: GitLab
+Contact: ~Infrastructure, ~Development in Mattermost and #rockylinux-infra, #rockylinux-devel in Libera IRC
Purpose: Users can subscribe and interact with various mail lists for the Rocky ecosystem
+Technology: Mailman 3 + Hyper Kitty
+Contact: ~Infrastructure in Mattermost and #rockylinux-infra in Libera IRC
| Name | +Mattermost Name | +IRC Name | +|
|---|---|---|---|
| Louis Abel | +label@rockylinux.org | +@nazunalika | +Sokel/label/Sombra | +
| Mustafa Gezen | +mustafa@rockylinux.org | +@mustafa | +mstg | +
| Skip Grube | +skip@rockylinux.org | +@skip77 | ++ |
| Sherif Nagy | +sherif@rockylinux.org | +@sherif | ++ |
| Pablo Greco | +pgreco@rockylinux.org | +@pgreco | +pgreco | +
| Neil Hanlon | +neil@resf.org | +@neil | +neil | +
| Taylor Goodwill | +tg@resf.org | +@tgo | +tg | +
The Rocky Linux Core Special Interest Group (SIG/Core) dedicates themselves to the development, building, management, production, and release of Rocky Linux for the Enterprise Linux community and the many users around the world. This group is a mixture of core Rocky Linux developers and infrastructure and the members of this group are also members of other groups within the Rocky Linux community (such as SIG/AltArch) as well as the Enterprise Linux community as a whole.
+SIG/Core strives to ensure a stable distribution is developed, built, tested, and provided to the community from the RESF as a compatible derivative of Red Hat Enterprise Linux. To achieve this goal, some of the things we do are:
+See the What We Do page for a more detailed explanation of our activities.
+There are various ways to get in touch with SIG/Core and provide help, assistance, or even just ideas that can benefit us or the entire community.
+Chat
+For a list of our members, see the Members page.
+SIG/Core is a mix of Release Engineering and Infrastructure members to ensure a high quality release of Rocky Linux as well as the uptime of the services provided to the community. The current members of this group are listed in the table below.
+| Role | Name | Email | Mattermost Name | IRC Name | |---------------------------------------------------|---------------------------------|------------------------ |-------------------|--------------------| +| Identity Management & Release Engineering Co-Lead | Louis Abel | label@rockylinux.org | @nazunalika | Sokel/label/Sombra | +| Release Engineering Co-Lead | Mustafa Gezen | mustafa@rockylinux.org | @mustafa | mstg | +| Release Engineering and Development | Skip Grube | skip@rockylinux.org | @skip77 | | +| Release Engineering and Development | Sherif Nagy | sherif@rockylinux.org | @sherif | | +| Release Engineering and Development | Pablo Greco | pgreco@rockylinux.org | @pgreco | pgreco | +| Infrastructure Lead | Neil Hanlon | neil@resf.org | @neil | neil | +| Infrastructure Lead | Taylor Goodwill | tg@resf.org | @tgo | tg |
+ +The Rocky Linux Core Special Interest Group (SIG/Core) dedicates themselves to the development, building, management, production, and release of Rocky Linux for the Enterprise Linux community and the many users around the world. This group is a mixture of core Rocky Linux developers and infrastructure and the members of this group are also members of other groups within the Rocky Linux community (such as SIG/AltArch) as well as the Enterprise Linux community as a whole.
"},{"location":"#mission","title":"Mission","text":"SIG/Core strives to ensure a stable distribution is developed, built, tested, and provided to the community from the RESF as a compatible derivative of Red Hat Enterprise Linux. To achieve this goal, some of the things we do are:
See the What We Do page for a more detailed explanation of our activities.
"},{"location":"#getting-in-touch-contributing","title":"Getting In Touch / Contributing","text":"There are various ways to get in touch with SIG/Core and provide help, assistance, or even just ideas that can benefit us or the entire community.
Chat
RESF SIG/Core Issue Tracker
For a list of our members, see the Members page.
"},{"location":"#resources-and-rocky-linux-policies","title":"Resources and Rocky Linux Policies","text":"SIG/Core is a mix of Release Engineering and Infrastructure members to ensure a high quality release of Rocky Linux as well as the uptime of the services provided to the community. The current members of this group are listed in the table below.
| Role | Name | Email | Mattermost Name | IRC Name | |---------------------------------------------------|---------------------------------|------------------------ |-------------------|--------------------| | Identity Management & Release Engineering Co-Lead | Louis Abel | label@rockylinux.org | @nazunalika | Sokel/label/Sombra | | Release Engineering Co-Lead | Mustafa Gezen | mustafa@rockylinux.org | @mustafa | mstg | | Release Engineering and Development | Skip Grube | skip@rockylinux.org | @skip77 | | | Release Engineering and Development | Sherif Nagy | sherif@rockylinux.org | @sherif | | | Release Engineering and Development | Pablo Greco | pgreco@rockylinux.org | @pgreco | pgreco | | Infrastructure Lead | Neil Hanlon | neil@resf.org | @neil | neil | | Infrastructure Lead | Taylor Goodwill | tg@resf.org | @tgo | tg |
"},{"location":"documentation/","title":"Composing Releases","text":"This section goes over at a high level how we compose releases for Rocky Linux. As most of our tools are home grown, we have made sure that the tools are open source and in our git services.
This page should serve as an idea of the steps we generally take and we hope that other projects out there who wish to also use our tools can make sure they can use them in this same way, whether they want to be an Enterprise Linux derivative or another project entirely.
"},{"location":"documentation/#build-system-and-tools","title":"Build System and Tools","text":"The tools in use for the distribution are in the table below.
Tool Maintainer Code Location srpmproc SIG/Core at RESF GitHub empanadas SIG/Core at RESF sig-core-toolkit Peridot SIG/Core at RESF GitHub MirrorManager 2 Fedora Project MirrorManager2For Rocky Linux to be build, we use Peridot as the build system and empanadas to \"compose\" the distribution. As we do not use Koji for Rocky Linux beyond version 9, pungi can no longer be used. Peridot instead takes pungi configuration data and comps and transforms them into a format it can understand. Empanadas then comes in to do the \"compose\" and sync all the repositories down.
Step by step, it looks like this:
In Parallel:
Lorax results are pulled down from an S3 bucket
Step by step, it looks like this:
URL: https://accounts.rockylinux.org
Purpose: Account Services maintains the accounts for almost all components of the Rocky ecosystem
Technology: Noggin used by Fedora Infrastructure
Contact: ~Infrastructure in Mattermost and #rockylinux-infra in Libera IRC
URL: https://git.resf.org
Purpose: General projects, code, and so on for the Rocky Enterprise Software Foundation.
Technology: Gitea
Contact: ~Infrastructure, ~Development in Mattermost and #rockylinux-infra, #rockylinux-devel in Libera IRC
URL: https://github.com/rocky-linux
Purpose: General purpose code, assets, and so on for Rocky Linux. Some content is mirrored to the RESF Git Service.
Technology: GitHub
Contact: ~Infrastructure, ~Development in Mattermost and #rockylinux-infra, #rockylinux-devel in Libera IRC
URL: https://git.rockylinux.org
Purpose: Packages and light code for the Rocky Linux distribution
Technology: GitLab
Contact: ~Infrastructure, ~Development in Mattermost and #rockylinux-infra, #rockylinux-devel in Libera IRC
URL: https://lists.resf.org
Purpose: Users can subscribe and interact with various mail lists for the Rocky ecosystem
Technology: Mailman 3 + Hyper Kitty
Contact: ~Infrastructure in Mattermost and #rockylinux-infra in Libera IRC
This page goes over empanadas, which is part of the SIG/Core toolkit. Empanadas assists SIG/Core is composing repositories, creating ISO's, creating images, and various other activities in Rocky Linux. It is also used for general testing and debugging of repositories and its metadata.
@label @neil Mattermost Channels ~Development"},{"location":"documentation/empanadas/#general-information","title":"General Information","text":"empanadas is a python project using poetry, containing various built-in modules with the goal to try to emulate the Fedora Project's pungi to an extent. While it is not perfect, it achieves the very basic goals of creating repositories, images and ISO's for consumption by the end user. It also has interactions with peridot, the build system used by the RESF to build the Rocky Linux distribution.
For performing syncs, it relies on the use of podman to perform syncing in a parallel fashion. This was done because it is not possible to run multiple dnf transactions at once on a single system and looping one repository at a time is not sustainable (nor fast).
"},{"location":"documentation/empanadas/#requirements","title":"Requirements","text":"fpart must be installed on the system (available in EPEL on EL systems)mock must be installed if building live images%rhel macro setAs of this writing, empanadas has the following abilities:
loraxxorriso for DVD and minimal imageslivemedia-creator and mockThe below is how to install empanadas from the development branch on a Fedora system.
% dnf install git podman fpart poetry mock -y\n% git clone https://git.resf.org/sig_core/toolkit.git -b devel\n% cd toolkit/iso/empanadas\n% poetry install\nDepending on how you are using empanadas will depend on how your configurations will be setup.
empanadas/common.pyempanadas/config/*.yamlempanadas/sig/*.yamlThese configuration files are delicate and can control a wide variety of the moving parts of empanadas. As these configurations are fairly massive, we recommend checking the reference guides for deeper details into configuring for base distribution or \"SIG\" content.
"},{"location":"documentation/empanadas/#using-empanadas","title":"Using Empanadas","text":"The most common way to use empanadas is to sync repositories from a peridot instance. This is performed upon each release or on each set of updates as they come from upstream. Below lists how to use empanadas, as well as the common options.
Note that for each of these commands, it is fully expected you are running poetry run in the root of empanadas.
URL: https://accounts.rockylinux.org
Purpose: Account Services maintains the accounts for almost all components of the Rocky ecosystem
Technology: Noggin used by Fedora Infrastructure
Contact: ~Infrastructure in Mattermost and #rockylinux-infra in Libera IRC
URL: https://git.resf.org
Purpose: General projects, code, and so on for the Rocky Enterprise Software Foundation.
Technology: Gitea
Contact: ~Infrastructure, ~Development in Mattermost and #rockylinux-infra, #rockylinux-devel in Libera IRC
URL: https://github.com/rocky-linux
Purpose: General purpose code, assets, and so on for Rocky Linux. Some content is mirrored to the RESF Git Service.
Technology: GitHub
Contact: ~Infrastructure, ~Development in Mattermost and #rockylinux-infra, #rockylinux-devel in Libera IRC
URL: https://git.rockylinux.org
Purpose: Packages and light code for the Rocky Linux distribution
Technology: GitLab
Contact: ~Infrastructure, ~Development in Mattermost and #rockylinux-infra, #rockylinux-devel in Libera IRC
URL: https://lists.resf.org
Purpose: Users can subscribe and interact with various mail lists for the Rocky ecosystem
Technology: Mailman 3 + Hyper Kitty
Contact: ~Infrastructure in Mattermost and #rockylinux-infra in Libera IRC
This page goes over the Peridot Build System and how SIG/Core utilizes it.
More to come.
"},{"location":"documentation/rebuild/","title":"Rebuild Version Bump","text":"In some cases, a package has to be rebuilt. A package may be rebuilt for these reasons:
This typically applies to packages being built from a given src subgroup. Packages pulled from upstream don't fall into this category in normal circumstances. In those cases, they receive .0.1 and so on as standalone rebuilds.
This section goes over the process of composing a release.
"},{"location":"documentation/references/","title":"References","text":"Use this section to locate reference configuration items for the toolkit.
"},{"location":"documentation/references/empanadas_common/","title":"Empanadas common.py Configuration","text":"The common.py configuration contains dictionaries and classes that dictate most of the functionality of empanadas.
type: Dictionary
"},{"location":"documentation/references/empanadas_common/#configrlmacro","title":"config.rlmacro","text":"type: String
required: True
description: Empanadas expects to run on an EL system. This is part of the general check up. It should not be hardcoded and use the rpm python module.
"},{"location":"documentation/references/empanadas_common/#configdist","title":"config.dist","text":"type: String
required: False
description: Was the original tag placed in mock configs. This combines el with the rpm python module expansion. This is no longer required. The option is still available for future use.
type: String
required: True
description: The architecture of the current running system. This is checked against the supported architectures in general release configurations. This should not be hardcoded.
"},{"location":"documentation/references/empanadas_common/#configdate_stamp","title":"config.date_stamp","text":"type: String
required: True
description: Date time stamp in the form of YYYYMMDD.HHMMSS. This should not be hardcoded.
"},{"location":"documentation/references/empanadas_common/#configcompose_root","title":"config.compose_root","text":"type: String
required: True
description: Root path of composes on the system running empanadas.
"},{"location":"documentation/references/empanadas_common/#configstaging_root","title":"config.staging_root","text":"type: String
required: False
description: For future use. Root path of staging repository location where content will be synced to.
"},{"location":"documentation/references/empanadas_common/#configproduction_root","title":"config.production_root","text":"type: String
required: False
description: For future use. Root path of production repository location where content will be synced to from staging.
"},{"location":"documentation/references/empanadas_common/#configcategory_stub","title":"config.category_stub","text":"type: String
required: True
description: For future use. Stub path that is appended to staging_root and production_root.
example: mirror/pub/rocky
type: String
required: True
description: For future use. Stub path that is appended to staging_root and production_root for SIG content.
example: mirror/pub/sig
type: String
required: True
description: URL to the base url's where the repositories live. This is typically to a peridot instance. This is supplemented by the configuration project_id parameter.
Note that this does not have to be a peridot instance. The combination of this value and project_id can be sufficient enough for empanadas to perform its work.
type: String
required: True
description: Hardcoded path to where ISO work is performed within a mock chroot. This is the default path created by mock and it is recommended not to change this.
example: /builddir
type: String
required: True
description: This is the container used to perform all operations in podman.
example: centos:stream9
type: String
required: True
description: Name of the distribution you are building or building for.
example: Rocky Linux
type: String
required: True
description: Short name of the distribution you are building or building for.
example: Rocky
type: Dictionary
required: True
description: Translates Linux architectures to golang architectures. Reserved for future use.
"},{"location":"documentation/references/empanadas_common/#configaws_region","title":"config.aws_region","text":"type: String
required: False
description: Region you are working in with AWS or onprem cloud that supports this variable.
example: us-east-2
type: String
required: False
description: Name of the S3-compatible bucket that is used to pull images from. Requires aws_region.
type: String
required: False
description: URL of the S3-compatible bucket that is used to pull images from.
"},{"location":"documentation/references/empanadas_common/#allowed_type_variants-items","title":"allowed_type_variants items","text":"type: Dictionary
description: Key value pairs of cloud or image variants. The value is either None or a list type.
config = {\n \"rlmacro\": rpm.expandMacro('%rhel'),\n \"dist\": 'el' + rpm.expandMacro('%rhel'),\n \"arch\": platform.machine(),\n \"date_stamp\": time.strftime(\"%Y%m%d.%H%M%S\", time.localtime()),\n \"compose_root\": \"/mnt/compose\",\n \"staging_root\": \"/mnt/repos-staging\",\n \"production_root\": \"/mnt/repos-production\",\n \"category_stub\": \"mirror/pub/rocky\",\n \"sig_category_stub\": \"mirror/pub/sig\",\n \"repo_base_url\": \"https://yumrepofs.build.resf.org/v1/projects\",\n \"mock_work_root\": \"/builddir\",\n \"container\": \"centos:stream9\",\n \"distname\": \"Rocky Linux\",\n \"shortname\": \"Rocky\",\n \"translators\": {\n \"x86_64\": \"amd64\",\n \"aarch64\": \"arm64\",\n \"ppc64le\": \"ppc64le\",\n \"s390x\": \"s390x\",\n \"i686\": \"386\"\n },\n \"aws_region\": \"us-east-2\",\n \"bucket\": \"resf-empanadas\",\n \"bucket_url\": \"https://resf-empanadas.s3.us-east-2.amazonaws.com\"\n}\n\nALLOWED_TYPE_VARIANTS = {\n \"Azure\": None,\n \"Container\": [\"Base\", \"Minimal\", \"UBI\"],\n \"EC2\": None,\n \"GenericCloud\": None,\n \"Vagrant\": [\"Libvirt\", \"Vbox\"],\n \"OCP\": None\n\n}\nEach file in empanads/config/ is a yaml file that contains configuration items for the distribution release version. The configuration can heavily dictate the functionality and what features are directly supported by empanadas when ran.
See the items below to see which options are mandatory and optional.
"},{"location":"documentation/references/empanadas_config/#config-items","title":"Config Items","text":""},{"location":"documentation/references/empanadas_config/#top-level","title":"Top Level","text":"The Top Level is the name of the profile and starts the YAML dictionary for the release. It is alphanumeric and accepts punctuation within reason. Common examples:
99-beta8-lookaheadtype: String
required: True
description: Needed for treeinfo and discinfo generation.
"},{"location":"documentation/references/empanadas_config/#revision","title":"revision","text":"type: String
required: True
description: Full version of a release
"},{"location":"documentation/references/empanadas_config/#rclvl","title":"rclvl","text":"type: String
required: True
description: Release Candidate or Beta descriptor. Sets names and versions with this descriptor if enabled.
"},{"location":"documentation/references/empanadas_config/#major","title":"major","text":"type: String
required: True
description: Major version of a release
"},{"location":"documentation/references/empanadas_config/#minor","title":"minor","text":"type: String
required: True
description: Minor version of a release
"},{"location":"documentation/references/empanadas_config/#profile","title":"profile","text":"type: String
required: True
description: Matches the top level of the release. This should not differ from the top level assignment.
"},{"location":"documentation/references/empanadas_config/#disttag","title":"disttag","text":"type: String
required: True
description: Sets the dist tag for mock configs.
"},{"location":"documentation/references/empanadas_config/#bugurl","title":"bugurl","text":"type: String
required: True
description: A URL to the bug tracker for this release or distribution.
"},{"location":"documentation/references/empanadas_config/#checksum","title":"checksum","text":"type: String
required: True
description: Checksum type. Used when generating checksum information for images.
"},{"location":"documentation/references/empanadas_config/#fedora_major","title":"fedora_major","text":"type: String
required: False
description: For future use with icicle.
"},{"location":"documentation/references/empanadas_config/#allowed_arches","title":"allowed_arches","text":"type: list
required: True
description: List of supported architectures for this release.
"},{"location":"documentation/references/empanadas_config/#provide_multilib","title":"provide_multilib","text":"type: boolean
required: True
description: Sets if architecture x86_64 will be multilib. It is recommended that this is set to True.
type: String
required: True
description: Appended to the base repo URL in common.py. For peridot, it is the project id that is generated for the project you are pulling from. It can be set to anything else if need be for non-peridot use.
"},{"location":"documentation/references/empanadas_config/#repo_symlinks","title":"repo_symlinks","text":"type: dict
required: False
description: For future use. Sets symlinks to repositories for backwards compatibility. Key value pairs only.
"},{"location":"documentation/references/empanadas_config/#renames","title":"renames","text":"type: dict
required: False
description: Renames a repository to the value set. For example, renaming all to devel. Set to {} if no renames are goign to occur.
type: list
required: True
description: List of repositories that will be synced/managed by empanadas.
"},{"location":"documentation/references/empanadas_config/#structure","title":"structure","text":"type: dict
required: True
description: Key value pairs of packages and repodata. These are appended appropriately during syncing and ISO actions. Setting these are mandatory.
type: dictionary
required: True if building ISO's and operating with lorax.
description: Controls how lorax and extra ISO's are built.
If are you not building images, set to {}
type: boolean
required: True
description: Dictates of xorrisofs is used to build images. Setting to false uses genisoimage. It is recommended that xorrisofs is used.
"},{"location":"documentation/references/empanadas_config/#iso_level","title":"iso_level","text":"type: boolean
required: True
description: Set to false if you are using xorrisofs. Can be set to true when using genisoimage.
"},{"location":"documentation/references/empanadas_config/#images","title":"images","text":"type: dict
required: True
description: Dictates the ISO images that will be made or the treeinfo that will be generated.
Note: The primary repository (for example, BaseOS) will need to be listed to ensure the treeinfo data is correctly generated. disc should be set to False and isoskip should be set to True. See the example section for an example.
type: boolean
required: True
description: This tells the iso builder if this will be a generated ISO.
"},{"location":"documentation/references/empanadas_config/#nameisoskip","title":"name.isoskip","text":"type: boolean
required: False
description: This tells the iso builder if this will be skipped, even if disc is set to True. Default is False.
type: string
required: True
description: Names the primary variant repository for the image. This is set in .treeinfo.
"},{"location":"documentation/references/empanadas_config/#namerepos","title":"name.repos","text":"type: list
required: True
description: Names of the repositories included in the image. This is added to .treeinfo.
"},{"location":"documentation/references/empanadas_config/#namevolname","title":"name.volname","text":"type: string
required: True
required value: dvd
description: This is required if building more than the DVD image. By default, the the name dvd is harcoded in the buildImage template.
type: dict
required: True if building lorax images.
description: Sets up lorax images and which repositories to use when building lorax images.
"},{"location":"documentation/references/empanadas_config/#loraxrepos","title":"lorax.repos","text":"type: list
required: True
description: List of repos that are used to pull packages to build the lorax images.
"},{"location":"documentation/references/empanadas_config/#loraxvariant","title":"lorax.variant","text":"type: string
required: True
description: Base repository for the release
"},{"location":"documentation/references/empanadas_config/#loraxlorax_removes","title":"lorax.lorax_removes","text":"type: list
required: False
description: Excludes packages that are not needed when lorax is running.
"},{"location":"documentation/references/empanadas_config/#loraxrequired_pkgs","title":"lorax.required_pkgs","text":"type: list
required: True
description: Required list of installed packages needed to build lorax images.
"},{"location":"documentation/references/empanadas_config/#livemap","title":"livemap","text":"type: dict
required: False
description: Dictates what live images are built and how they are built.
"},{"location":"documentation/references/empanadas_config/#livemapgit_repo","title":"livemap.git_repo","text":"type: string
required: True
description: The git repository URL where the kickstarts live
"},{"location":"documentation/references/empanadas_config/#livemapbranch","title":"livemap.branch","text":"type: string
required: True
description: The branch being used for the kickstarts
"},{"location":"documentation/references/empanadas_config/#livemapksentry","title":"livemap.ksentry","text":"type: dict
required: True
description: Key value pairs of the live images being created. Key being the name of the live image, value being the kickstart name/path.
"},{"location":"documentation/references/empanadas_config/#livemapallowed_arches","title":"livemap.allowed_arches","text":"type: list
required: True
description: List of allowed architectures that will build for the live images.
"},{"location":"documentation/references/empanadas_config/#livemaprequired_pkgs","title":"livemap.required_pkgs","text":"type: list
required: True
description: Required list of packages needed to build the live images.
"},{"location":"documentation/references/empanadas_config/#cloudimages","title":"cloudimages","text":"type: dict
required: False
description: Cloud related settings.
Set to {} if not needed.
type: dict
required: True
description: Cloud images that will be generated and in a bucket to be pulled, and their format.
"},{"location":"documentation/references/empanadas_config/#cloudimagesimagesname","title":"cloudimages.images.name","text":"type: dict
required: True
description: Name of the cloud image being pulled.
Accepted key value options:
format, which is raw, qcow2, vhd, tar.xzvariants, which is a listprimary_variant, which symlinks to the \"primary\" variant in the variant listtype: dict
required: True
description: Repoclosure settings. These settings are absolutely required when doing full syncs and need to check repositories for consistency.
"},{"location":"documentation/references/empanadas_config/#repoclosure_maparches","title":"repoclosure_map.arches","text":"type: dict
required: True
description: For each architecture (key), dnf switches/settings that dictate how repoclosure will check for consistency (value, string).
example: x86_64: '--forcearch=x86_64 --arch=x86_64 --arch=athlon --arch=i686 --arch=i586 --arch=i486 --arch=i386 --arch=noarch'
type: dict
required: True
description: For each repository that is pulled for a given release(key), repositories that will be included in the repoclosure check. A repository that only checks against itself must have a value of [].
type: dict
required: True
description: Extra files settings and where they come from. Git repositories are the only supported method.
"},{"location":"documentation/references/empanadas_config/#extra_filesgit_repo","title":"extra_files.git_repo","text":"type: string
required: True
description: URL to the git repository with the extra files.
"},{"location":"documentation/references/empanadas_config/#extra_filesgit_raw_path","title":"extra_files.git_raw_path","text":"type: string
required: True
description: URL to the git repository with the extra files, but the \"raw\" url form.
example: git_raw_path: 'https://git.rockylinux.org/staging/src/rocky-release/-/raw/r9/'
type: string
required: True
description: Branch where the extra files are pulled from.
"},{"location":"documentation/references/empanadas_config/#extra_filesgpg","title":"extra_files.gpg","text":"type: dict
required: True
description: For each gpg key type (key), the relative path to the key in the git repository (value).
These keys help set up the repository configuration when doing syncs.
By default, the RepoSync class sets stable as the gpgkey that is used.
type: list
required: True
description: List of files from the git repository that will be used as \"extra\" files and placed in the repositories and available to mirrors and will appear on ISO images if applicable.
"},{"location":"documentation/references/empanadas_config/#reference-example","title":"Reference Example","text":"---\n'9':\n fullname: 'Rocky Linux 9.0'\n revision: '9.0'\n rclvl: 'RC2'\n major: '9'\n minor: '0'\n profile: '9'\n disttag: 'el9'\n bugurl: 'https://bugs.rockylinux.org'\n checksum: 'sha256'\n fedora_major: '20'\n allowed_arches:\n - x86_64\n - aarch64\n - ppc64le\n - s390x\n provide_multilib: True\n project_id: '55b17281-bc54-4929-8aca-a8a11d628738'\n repo_symlinks:\n NFV: 'nfv'\n renames:\n all: 'devel'\n all_repos:\n - 'all'\n - 'BaseOS'\n - 'AppStream'\n - 'CRB'\n - 'HighAvailability'\n - 'ResilientStorage'\n - 'RT'\n - 'NFV'\n - 'SAP'\n - 'SAPHANA'\n - 'extras'\n - 'plus'\n structure:\n packages: 'os/Packages'\n repodata: 'os/repodata'\n iso_map:\n xorrisofs: True\n iso_level: False\n images:\n dvd:\n disc: True\n variant: 'AppStream'\n repos:\n - 'BaseOS'\n - 'AppStream'\n minimal:\n disc: True\n isoskip: True\n repos:\n - 'minimal'\n - 'BaseOS'\n variant: 'minimal'\n volname: 'dvd'\n BaseOS:\n disc: False\n isoskip: True\n variant: 'BaseOS'\n repos:\n - 'BaseOS'\n - 'AppStream'\n lorax:\n repos:\n - 'BaseOS'\n - 'AppStream'\n variant: 'BaseOS'\n lorax_removes:\n - 'libreport-rhel-anaconda-bugzilla'\n required_pkgs:\n - 'lorax'\n - 'genisoimage'\n - 'isomd5sum'\n - 'lorax-templates-rhel'\n - 'lorax-templates-generic'\n - 'xorriso'\n cloudimages:\n images:\n EC2:\n format: raw\n GenericCloud:\n format: qcow2\n livemap:\n git_repo: 'https://git.resf.org/sig_core/kickstarts.git'\n branch: 'r9'\n ksentry:\n Workstation: rocky-live-workstation.ks\n Workstation-Lite: rocky-live-workstation-lite.ks\n XFCE: rocky-live-xfce.ks\n KDE: rocky-live-kde.ks\n MATE: rocky-live-mate.ks\n allowed_arches:\n - x86_64\n - aarch64\n required_pkgs:\n - 'lorax-lmc-novirt'\n - 'vim-minimal'\n - 'pykickstart'\n - 'git'\n variantmap:\n git_repo: 'https://git.rockylinux.org/rocky/pungi-rocky.git'\n branch: 'r9'\n git_raw_path: 'https://git.rockylinux.org/rocky/pungi-rocky/-/raw/r9/'\n repoclosure_map:\n arches:\n x86_64: '--forcearch=x86_64 --arch=x86_64 --arch=athlon --arch=i686 --arch=i586 --arch=i486 --arch=i386 --arch=noarch'\n aarch64: '--forcearch=aarch64 --arch=aarch64 --arch=noarch'\n ppc64le: '--forcearch=ppc64le --arch=ppc64le --arch=noarch'\n s390x: '--forcearch=s390x --arch=s390x --arch=noarch'\n repos:\n devel: []\n BaseOS: []\n AppStream:\n - BaseOS\n CRB:\n - BaseOS\n - AppStream\n HighAvailability:\n - BaseOS\n - AppStream\n ResilientStorage:\n - BaseOS\n - AppStream\n RT:\n - BaseOS\n - AppStream\n NFV:\n - BaseOS\n - AppStream\n SAP:\n - BaseOS\n - AppStream\n - HighAvailability\n SAPHANA:\n - BaseOS\n - AppStream\n - HighAvailability\n extra_files:\n git_repo: 'https://git.rockylinux.org/staging/src/rocky-release.git'\n git_raw_path: 'https://git.rockylinux.org/staging/src/rocky-release/-/raw/r9/'\n branch: 'r9'\n gpg:\n stable: 'SOURCES/RPM-GPG-KEY-Rocky-9'\n testing: 'SOURCES/RPM-GPG-KEY-Rocky-9-Testing'\n list:\n - 'SOURCES/Contributors'\n - 'SOURCES/COMMUNITY-CHARTER'\n - 'SOURCES/EULA'\n - 'SOURCES/LICENSE'\n - 'SOURCES/RPM-GPG-KEY-Rocky-9'\n - 'SOURCES/RPM-GPG-KEY-Rocky-9-Testing'\n...\nEach file in empanads/sig/ is a yaml file that contains configuration items for the distribution release version. The configuration determines the structure of the SIG repositories synced from Peridot or a given repo.
Note that a release profile (for a major version) is still required for this sync to work.
See the items below to see which options are mandatory and optional.
"},{"location":"documentation/references/empanadas_sig_config/#config-items","title":"Config Items","text":""},{"location":"documentation/references/empanadas_sig_config/#reference-example","title":"Reference Example","text":""},{"location":"include/resources_bottom/","title":"Resources bottom","text":"Resources Account ServicesGit (RESF Git Service)Git (Rocky Linux GitHub)Git (Rocky Linux GitLab)Mail ListsContactsURL: https://accounts.rockylinux.org
Purpose: Account Services maintains the accounts for almost all components of the Rocky ecosystem
Technology: Noggin used by Fedora Infrastructure
Contact: ~Infrastructure in Mattermost and #rockylinux-infra in Libera IRC
URL: https://git.resf.org
Purpose: General projects, code, and so on for the Rocky Enterprise Software Foundation.
Technology: Gitea
Contact: ~Infrastructure, ~Development in Mattermost and #rockylinux-infra, #rockylinux-devel in Libera IRC
URL: https://github.com/rocky-linux
Purpose: General purpose code, assets, and so on for Rocky Linux. Some content is mirrored to the RESF Git Service.
Technology: GitHub
Contact: ~Infrastructure, ~Development in Mattermost and #rockylinux-infra, #rockylinux-devel in Libera IRC
URL: https://git.rockylinux.org
Purpose: Packages and light code for the Rocky Linux distribution
Technology: GitLab
Contact: ~Infrastructure, ~Development in Mattermost and #rockylinux-infra, #rockylinux-devel in Libera IRC
URL: https://lists.resf.org
Purpose: Users can subscribe and interact with various mail lists for the Rocky ecosystem
Technology: Mailman 3 + Hyper Kitty
Contact: ~Infrastructure in Mattermost and #rockylinux-infra in Libera IRC
This section goes over the various SOP's for SIG/Core. Please use the menu items to find the various pages of interest.
Resources Account ServicesGit (RESF Git Service)Git (Rocky Linux GitHub)Git (Rocky Linux GitLab)Mail ListsContactsURL: https://accounts.rockylinux.org
Purpose: Account Services maintains the accounts for almost all components of the Rocky ecosystem
Technology: Noggin used by Fedora Infrastructure
Contact: ~Infrastructure in Mattermost and #rockylinux-infra in Libera IRC
URL: https://git.resf.org
Purpose: General projects, code, and so on for the Rocky Enterprise Software Foundation.
Technology: Gitea
Contact: ~Infrastructure, ~Development in Mattermost and #rockylinux-infra, #rockylinux-devel in Libera IRC
URL: https://github.com/rocky-linux
Purpose: General purpose code, assets, and so on for Rocky Linux. Some content is mirrored to the RESF Git Service.
Technology: GitHub
Contact: ~Infrastructure, ~Development in Mattermost and #rockylinux-infra, #rockylinux-devel in Libera IRC
URL: https://git.rockylinux.org
Purpose: Packages and light code for the Rocky Linux distribution
Technology: GitLab
Contact: ~Infrastructure, ~Development in Mattermost and #rockylinux-infra, #rockylinux-devel in Libera IRC
URL: https://lists.resf.org
Purpose: Users can subscribe and interact with various mail lists for the Rocky ecosystem
Technology: Mailman 3 + Hyper Kitty
Contact: ~Infrastructure in Mattermost and #rockylinux-infra in Libera IRC
This SOP contains most if not all the information needed for SIG/Core to maintain and operate Mirror Manager for Rocky Linux.
"},{"location":"sop/sop_mirrormanager2/#contact-information","title":"Contact Information","text":"Owner SIG/Core (Release Engineering & Infrastructure) Email Contact infrastructure@rockylinux.org Email Contact releng@rockylinux.org Mattermost Contacts@label @neil @tgo Mattermost Channels ~Infrastructure"},{"location":"sop/sop_mirrormanager2/#introduction","title":"Introduction","text":"So you made a bad decision and now have to do things to Mirror Manager. Good luck.
"},{"location":"sop/sop_mirrormanager2/#pieces","title":"Pieces","text":"Item Runs on... Software Mirrorlist Server mirrormanager001 https://github.com/adrianreber/mirrorlist-server/ Mirror Manager 2 mirrormanager001 https://github.com/fedora-infra/mirrormanager2"},{"location":"sop/sop_mirrormanager2/#mirrorlist-server","title":"Mirrorlist Server","text":"This runs two (2) instances. Apache/httpd is configured to send /mirrorlist to one and /debuglist to the other.
Every fifteen (15) minutes: Mirrorlist cache is regenerated
Every twenty (20) minutes: Service hosting /mirrorlist is restarted
/debuglist is restartedNote that the timing for the restart of the mirror list instances are arbitrary.
"},{"location":"sop/sop_mirrormanager2/#mirror-manager-2","title":"Mirror Manager 2","text":"This is a uwsgi service fronted by an apache/httpd instance. This is responsible for everything else that is not /mirrorlist or /debuglist. This allows the mirror managers to, well, manage their mirrors.
Fastly sits in front of mirror manager. VPN is required to access the /admin endpoints.
If the backend of the CDN is down, it will attempt to guess what the user wanted to access and spit out a result on the dl.rockylinux.org website. For example, a request for AppStream-8 and x86_64 will result in a AppStream/x86_64/os directory on dl.rockylinux.org. Note that this isn't perfect, but it helps in potential down time or patching.
Fastly -> www firewall -> mirrormanager server\nIn reality, the flow is a lot more complex, and a diagram should be created to map it out in a more user-friendly manner (@TODO)
User -> Fastly -> AWS NLB over TLS, passthru -> www firewall cluster (decrypt TLS) -> mirrormanager server (Rocky CA TLS)\nBelow are a list of possible tasks to take with mirror manager, depending on the scenario.
"},{"location":"sop/sop_mirrormanager2/#new-release","title":"New Release","text":"For the following steps, the following must be completed:
Disable mirrormanager user cronjobs
Update the database with the new content. This is run on a schedule normally (see previous section) but can be done manually.
a. As the mirror manager user, run the following:
/opt/mirrormanager/scan-primary-mirror-0.4.2/target/debug/scan-primary-mirror --debug --config $HOME/scan-primary-mirror.toml --category 'Rocky Linux'\n/opt/mirrormanager/scan-primary-mirror-0.4.2/target/debug/scan-primary-mirror --debug --config $HOME/scan-primary-mirror.toml --category 'Rocky Linux SIGs'\nUpdate the redirects for $reponame-$releasever
a. Use psql to mirrormanager server: psql -U mirrormanager -W -h mirrormanager_db_host mirrormanager_db b. Confirm that all three columns are filled and that the second and third columns are identical: select rr.from_repo AS \"From Repo\", rr.to_repo AS \"To Repo\", r.prefix AS \"Target Repo\" FROM repository_redirect AS rr LEFT JOIN repository AS r ON rr.to_repo = r.prefix GROUP BY r.prefix, rr.to_repo, rr.from_repo ORDER BY r.prefix ASC; c. Change the majorversion redirects to point to the new point release, for example: update repository_redirect set to_repo = regexp_replace(to_repo, '9\\.0', '9.1') where from_repo ~ '(\\w+)-9';
Generate the mirrorlist cache and restart the debuglist and verify.
Once the bitflip is initiated, restart mirrorlist and reenable all cronjobs.
"},{"location":"sop/sop_mirrormanager2/#out-of-date-mirrors","title":"Out-of-date Mirrors","text":"shasum=$(curl https://dl.rockylinux.org/pub/rocky/9.0/BaseOS/x86_64/os/repodata/repomd.xml | sha256sum)tail -latr /var/log/mirrormanager/propagation/rocky-9.0-BaseOS-x86_64_propagation.log.*`\n\nexport VER=9.0\nawk -v shasum=$(curl -s https://dl.rockylinux.org/pub/rocky/$VER/BaseOS/x86_64/os/repodata/repomd.xml | sha256sum | awk '{print $1}') -F'::' '{split($0,data,\":\")} {if ($4 != shasum) {print data[5], data[6], $2, $7}}' < $(find /var/log/mirrormanager/propagation/ -name \"rocky-${VER}-BaseOS-x86_64_propagation.log*\" -mtime -1 | tail -1)'\nThis will generate a table. You can take the IDs in the first column and use the database to disable them by ID (table name: hosts) or go to https://mirrors.rockylinux.org/mirrormanager/host/ID and uncheck 'User active'.
Users can change user active, but they cannot change admin active. It is better to flip user active in this case.
Admins can also view https://mirrors.rockylinux.org/mirrormanager/admin/all_sites if necessary.
Example of table columns:
[mirrormanager@ord1-prod-mirrormanager001 propagation]$ awk -v shasum=$(curl -s https://dl.rockylinux.org/pub/rocky/9.0/BaseOS/x86_64/os/repodata/repomd.xml | sha256sum | awk '{print $1}') -F'::' '{split($0,data,\":\")} {if ($4 != shasum) {print data[5], data[6], $2, $7}}' < rocky-9.0-BaseOS-x86_64_propagation.log.1660611632 | column -t\n164 mirror.host.ag http://mirror.host.ag/rocky/9.0/BaseOS/x86_64/os/repodata/repomd.xml 404\n173 rocky.centos-repo.net http://rocky.centos-repo.net/9.0/BaseOS/x86_64/os/repodata/repomd.xml 403\n92 rocky.mirror.co.ge http://rocky.mirror.co.ge/9.0/BaseOS/x86_64/os/repodata/repomd.xml 404\n289 mirror.vsys.host http://mirror.vsys.host/rockylinux/9.0/BaseOS/x86_64/os/repodata/repomd.xml 404\n269 mirrors.rackbud.com http://mirrors.rackbud.com/rocky/9.0/BaseOS/x86_64/os/repodata/repomd.xml 200\n295 mirror.ps.kz http://mirror.ps.kz/rocky/9.0/BaseOS/x86_64/os/repodata/repomd.xml 200\n114 mirror.liteserver.nl http://rockylinux.mirror.liteserver.nl/9.0/BaseOS/x86_64/os/repodata/repomd.xml 200\n275 mirror.upsi.edu.my http://mirror.upsi.edu.my/rocky/9.0/BaseOS/x86_64/os/repodata/repomd.xml 200\n190 mirror.kku.ac.th http://mirror.kku.ac.th/rocky-linux/9.0/BaseOS/x86_64/os/repodata/repomd.xml 404\n292 mirrors.cat.pdx.edu http://mirrors.cat.pdx.edu/rocky/9.0/BaseOS/x86_64/os/repodata/repomd.xml 200\n370 mirrors.gbnetwork.com http://mirrors.gbnetwork.com/rocky/9.0/BaseOS/x86_64/os/repodata/repomd.xml 404\n308 mirror.ihost.md http://mirror.ihost.md/rockylinux/9.0/BaseOS/x86_64/os/repodata/repomd.xml 404\n87 mirror.freedif.org http://mirror.freedif.org/Rocky/9.0/BaseOS/x86_64/os/repodata/repomd.xml 404\n194 mirrors.bestthaihost.com http://mirrors.bestthaihost.com/rocky/9.0/BaseOS/x86_64/os/repodata/repomd.xml 404\n30 mirror.admax.se http://mirror.admax.se/rocky/9.0/BaseOS/x86_64/os/repodata/repomd.xml 200\n195 mirror.uepg.br http://mirror.uepg.br/rocky/9.0/BaseOS/x86_64/os/repodata/repomd.xml 404\n247 mirrors.ipserverone.com http://mirrors.ipserverone.com/rocky/9.0/BaseOS/x86_64/os/repodata/repomd.xml 404'\nThis SOP contains all the steps required by SIG/Core (a mix of Release Engineering and Infrastructure) to perform releases of all Rocky Linux versions. Work is in all collaboration within the entire group of engineerings.
"},{"location":"sop/sop_release/#contact-information","title":"Contact Information","text":"Owner SIG/Core (Release Engineering & Infrastructure) Email Contact infrastructure@rockylinux.org Email Contact releng@rockylinux.org Mattermost Contacts@label @neil @tgo @skip77 @mustafa @sherif @pgreco Mattermost Channels ~Infrastructure"},{"location":"sop/sop_release/#preparation","title":"Preparation","text":""},{"location":"sop/sop_release/#notes-about-release-day","title":"Notes about Release Day","text":"Within a minimum of two (2) days, the following should be true:
Torrents should be setup. All files can be synced with the seed box(es) but not yet published. The data should be verified using sha256sum and compared to the CHECKSUM files provided with the files.
Website should be ready (typically with an open PR in github). The content should be verified that the design and content are correct and finalized.
Enough mirrors should be setup. This essentially means that all content for a release should be synced to our primary mirror with the executable bit turned off, and the content should also be hard linked. In theory, mirror manager can be queried to verify if mirrors are or appear to be in sync.
Within a minimum of one (1) to two (2) days, the following should be true:
Updates should be completed in the build system, and verified in staging.
Updates should be sent to production and file lists updated to allow mirrors to sync.
Ensure the SIG/Core Checklist is read thoroughly and executed as listed.
"},{"location":"sop/sop_release/#release-day","title":"Release Day","text":""},{"location":"sop/sop_release/#priorities","title":"Priorities","text":"During release day, these should be verified/completed in order:
Website - The primary website and user landing at rockylinux.org should allow the user to efficiently click through to a download link of an ISO, image, or torrent. It must be kept up.
Torrent - The seed box(es) should be primed and ready to go for users downloading via torrent.
Release Notes & Documentation - The release notes are often on the same website as the documentation. The main website and where applicable in the docs should refer to the Release Notes of Rocky Linux.
Wiki - If applicable, the necessary changes and resources should be available for a release. In particular, if a major release has new repos, changed repo names, this should be documented.
Everything else!
Mirror Manager
Pass image to Testing Team for final validation
Mirror Manager
Pass image to Testing Team for validation
Mirror Manager
Pass image to Testing Team for final validation
URL: https://accounts.rockylinux.org
Purpose: Account Services maintains the accounts for almost all components of the Rocky ecosystem
Technology: Noggin used by Fedora Infrastructure
Contact: ~Infrastructure in Mattermost and #rockylinux-infra in Libera IRC
URL: https://git.resf.org
Purpose: General projects, code, and so on for the Rocky Enterprise Software Foundation.
Technology: Gitea
Contact: ~Infrastructure, ~Development in Mattermost and #rockylinux-infra, #rockylinux-devel in Libera IRC
URL: https://github.com/rocky-linux
Purpose: General purpose code, assets, and so on for Rocky Linux. Some content is mirrored to the RESF Git Service.
Technology: GitHub
Contact: ~Infrastructure, ~Development in Mattermost and #rockylinux-infra, #rockylinux-devel in Libera IRC
URL: https://git.rockylinux.org
Purpose: Packages and light code for the Rocky Linux distribution
Technology: GitLab
Contact: ~Infrastructure, ~Development in Mattermost and #rockylinux-infra, #rockylinux-devel in Libera IRC
URL: https://lists.resf.org
Purpose: Users can subscribe and interact with various mail lists for the Rocky ecosystem
Technology: Mailman 3 + Hyper Kitty
Contact: ~Infrastructure in Mattermost and #rockylinux-infra in Libera IRC
This SOP contains general checklists required by SIG/Core to prepare and plan for the upcoming release. This work, in general, is required to be done on a routine basis, even months out before the next major or minor release, as it requires monitoring of upstream's (CentOS Stream) work to ensure Rocky Linux will remain ready and compatible with Red Hat Enterprise Linux.
"},{"location":"sop/sop_upstream_prep_checklist/#contact-information","title":"Contact Information","text":"Owner SIG/Core (Release Engineering & Infrastructure) Email Contact infrastructure@rockylinux.org Email Contact releng@rockylinux.org Mattermost Contacts@label @neil @tgo @skip77 @mustafa @sherif @pgreco Mattermost Channels ~Infrastructure"},{"location":"sop/sop_upstream_prep_checklist/#general-upstream-monitoring","title":"General Upstream Monitoring","text":"It is expected to monitor the following repositories upstream, as these will indicate what is coming up for a given major or point release. These repositories are found at the Red Hat gitlab.
These repositories can be monitored by setting to \"all activity\" on the bell icon.
Upon changes to the upstream repositories, SIG/Core member should analyze the changes and apply the same to the lookahead branches:
rocky-release
rocky-logos
pungi-rocky
sync-from-upstreamperidot-rocky
comps
sync-from-upstreamrocky-module-defaults
sync-from-upstreamRepositories that generally track for LookAhead and Beta releases will flow downward to the stable branch. For example:
* rXs / rXlh\n |\n |----> rX-beta\n |\n |----> rX\nThis applies to any specific rocky repo, such as comps, pungi, peridot-config, and so on. As it is expected some repos will deviate in commit history, it is OK to force push, under the assumption that changes made in the lower branch exists in the upper branch. That way you can avoid changes/functionality being reverted on accident.
"},{"location":"sop/sop_upstream_prep_checklist/#general-package-patching","title":"General Package Patching","text":"There are packages that are patched typically for the purpose of debranding. List of patched packages are typically maintained in a metadata repository. The obvious ones are listed below and should be monitored and maintained properly:
This section goes over the various SOP's for SIG/Core. Please use the menu items +to find the various pages of interest.
+URL: https://accounts.rockylinux.org
+Purpose: Account Services maintains the accounts for almost all components of the Rocky ecosystem
+Technology: Noggin used by Fedora Infrastructure
+Contact: ~Infrastructure in Mattermost and #rockylinux-infra in Libera IRC
URL: https://git.resf.org
+Purpose: General projects, code, and so on for the Rocky Enterprise Software Foundation.
+Technology: Gitea
+Contact: ~Infrastructure, ~Development in Mattermost and #rockylinux-infra, #rockylinux-devel in Libera IRC
URL: https://github.com/rocky-linux
+Purpose: General purpose code, assets, and so on for Rocky Linux. Some content is mirrored to the RESF Git Service.
+Technology: GitHub
+Contact: ~Infrastructure, ~Development in Mattermost and #rockylinux-infra, #rockylinux-devel in Libera IRC
URL: https://git.rockylinux.org
+Purpose: Packages and light code for the Rocky Linux distribution
+Technology: GitLab
+Contact: ~Infrastructure, ~Development in Mattermost and #rockylinux-infra, #rockylinux-devel in Libera IRC
Purpose: Users can subscribe and interact with various mail lists for the Rocky ecosystem
+Technology: Mailman 3 + Hyper Kitty
+Contact: ~Infrastructure in Mattermost and #rockylinux-infra in Libera IRC
| Name | +Mattermost Name | +IRC Name | +|
|---|---|---|---|
| Louis Abel | +label@rockylinux.org | +@nazunalika | +Sokel/label/Sombra | +
| Mustafa Gezen | +mustafa@rockylinux.org | +@mustafa | +mstg | +
| Skip Grube | +skip@rockylinux.org | +@skip77 | ++ |
| Sherif Nagy | +sherif@rockylinux.org | +@sherif | ++ |
| Pablo Greco | +pgreco@rockylinux.org | +@pgreco | +pgreco | +
| Neil Hanlon | +neil@resf.org | +@neil | +neil | +
| Taylor Goodwill | +tg@resf.org | +@tgo | +tg | +
This SOP contains most if not all the information needed for SIG/Core to +maintain and operate Mirror Manager for Rocky Linux.
+| + | + |
|---|---|
| Owner | +SIG/Core (Release Engineering & Infrastructure) | +
| Email Contact | +infrastructure@rockylinux.org | +
| Email Contact | +releng@rockylinux.org | +
| Mattermost Contacts | +@label @neil @tgo |
+
| Mattermost Channels | +~Infrastructure |
+
So you made a bad decision and now have to do things to Mirror Manager. Good +luck.
+| Item | +Runs on... | +Software | +
|---|---|---|
| Mirrorlist Server | +mirrormanager001 | +https://github.com/adrianreber/mirrorlist-server/ | +
| Mirror Manager 2 | +mirrormanager001 | +https://github.com/fedora-infra/mirrormanager2 | +
This runs two (2) instances. Apache/httpd is configured to send /mirrorlist
+to one and /debuglist to the other.
Every fifteen (15) minutes: Mirrorlist cache is regenerated
+Every twenty (20) minutes: Service hosting /mirrorlist is restarted
/debuglist is restartedNote that the timing for the restart of the mirror list instances are arbitrary.
+This is a uwsgi service fronted by an apache/httpd instance. This is responsible
+for everything else that is not /mirrorlist or /debuglist. This allows the
+mirror managers to, well, manage their mirrors.
Fastly sits in front of mirror manager. VPN is required to access the /admin endpoints.
If the backend of the CDN is down, it will attempt to guess what the user wanted to access and spit out a result on the dl.rockylinux.org website. For example, a request for AppStream-8 and x86_64 will result in a AppStream/x86_64/os directory on dl.rockylinux.org. Note that this isn't perfect, but it helps in potential down time or patching.
Fastly -> www firewall -> mirrormanager server
+In reality, the flow is a lot more complex, and a diagram should be created to map it out in a more user-friendly manner (@TODO)
+User -> Fastly -> AWS NLB over TLS, passthru -> www firewall cluster (decrypt TLS) -> mirrormanager server (Rocky CA TLS)
+Below are a list of possible tasks to take with mirror manager, depending on the scenario.
+For the following steps, the following must be completed:
+Disable mirrormanager user cronjobs
+Update the database with the new content. This is run on a schedule normally (see previous section) but can be done manually.
+a. As the mirror manager user, run the following:
+/opt/mirrormanager/scan-primary-mirror-0.4.2/target/debug/scan-primary-mirror --debug --config $HOME/scan-primary-mirror.toml --category 'Rocky Linux'
+/opt/mirrormanager/scan-primary-mirror-0.4.2/target/debug/scan-primary-mirror --debug --config $HOME/scan-primary-mirror.toml --category 'Rocky Linux SIGs'
+Update the redirects for $reponame-$releasever
a. Use psql to mirrormanager server: psql -U mirrormanager -W -h mirrormanager_db_host mirrormanager_db
+b. Confirm that all three columns are filled and that the second and third columns are identical: select rr.from_repo AS "From Repo", rr.to_repo AS "To Repo", r.prefix AS "Target Repo" FROM repository_redirect AS rr LEFT JOIN repository AS r ON rr.to_repo = r.prefix GROUP BY r.prefix, rr.to_repo, rr.from_repo ORDER BY r.prefix ASC;
+c. Change the majorversion redirects to point to the new point release, for example: update repository_redirect set to_repo = regexp_replace(to_repo, '9\.0', '9.1') where from_repo ~ '(\w+)-9';
Generate the mirrorlist cache and restart the debuglist and verify.
+Once the bitflip is initiated, restart mirrorlist and reenable all cronjobs.
+shasum=$(curl https://dl.rockylinux.org/pub/rocky/9.0/BaseOS/x86_64/os/repodata/repomd.xml | sha256sum)tail -latr /var/log/mirrormanager/propagation/rocky-9.0-BaseOS-x86_64_propagation.log.*`
+
+export VER=9.0
+awk -v shasum=$(curl -s https://dl.rockylinux.org/pub/rocky/$VER/BaseOS/x86_64/os/repodata/repomd.xml | sha256sum | awk '{print $1}') -F'::' '{split($0,data,":")} {if ($4 != shasum) {print data[5], data[6], $2, $7}}' < $(find /var/log/mirrormanager/propagation/ -name "rocky-${VER}-BaseOS-x86_64_propagation.log*" -mtime -1 | tail -1)'
+This will generate a table. You can take the IDs in the first column and use the database to disable them by ID (table name: hosts) or go to https://mirrors.rockylinux.org/mirrormanager/host/ID and uncheck 'User active'.
+Users can change user active, but they cannot change admin active. It is better to flip user active in this case.
+Admins can also view https://mirrors.rockylinux.org/mirrormanager/admin/all_sites if necessary.
+Example of table columns:
+[mirrormanager@ord1-prod-mirrormanager001 propagation]$ awk -v shasum=$(curl -s https://dl.rockylinux.org/pub/rocky/9.0/BaseOS/x86_64/os/repodata/repomd.xml | sha256sum | awk '{print $1}') -F'::' '{split($0,data,":")} {if ($4 != shasum) {print data[5], data[6], $2, $7}}' < rocky-9.0-BaseOS-x86_64_propagation.log.1660611632 | column -t
+164 mirror.host.ag http://mirror.host.ag/rocky/9.0/BaseOS/x86_64/os/repodata/repomd.xml 404
+173 rocky.centos-repo.net http://rocky.centos-repo.net/9.0/BaseOS/x86_64/os/repodata/repomd.xml 403
+92 rocky.mirror.co.ge http://rocky.mirror.co.ge/9.0/BaseOS/x86_64/os/repodata/repomd.xml 404
+289 mirror.vsys.host http://mirror.vsys.host/rockylinux/9.0/BaseOS/x86_64/os/repodata/repomd.xml 404
+269 mirrors.rackbud.com http://mirrors.rackbud.com/rocky/9.0/BaseOS/x86_64/os/repodata/repomd.xml 200
+295 mirror.ps.kz http://mirror.ps.kz/rocky/9.0/BaseOS/x86_64/os/repodata/repomd.xml 200
+114 mirror.liteserver.nl http://rockylinux.mirror.liteserver.nl/9.0/BaseOS/x86_64/os/repodata/repomd.xml 200
+275 mirror.upsi.edu.my http://mirror.upsi.edu.my/rocky/9.0/BaseOS/x86_64/os/repodata/repomd.xml 200
+190 mirror.kku.ac.th http://mirror.kku.ac.th/rocky-linux/9.0/BaseOS/x86_64/os/repodata/repomd.xml 404
+292 mirrors.cat.pdx.edu http://mirrors.cat.pdx.edu/rocky/9.0/BaseOS/x86_64/os/repodata/repomd.xml 200
+370 mirrors.gbnetwork.com http://mirrors.gbnetwork.com/rocky/9.0/BaseOS/x86_64/os/repodata/repomd.xml 404
+308 mirror.ihost.md http://mirror.ihost.md/rockylinux/9.0/BaseOS/x86_64/os/repodata/repomd.xml 404
+87 mirror.freedif.org http://mirror.freedif.org/Rocky/9.0/BaseOS/x86_64/os/repodata/repomd.xml 404
+194 mirrors.bestthaihost.com http://mirrors.bestthaihost.com/rocky/9.0/BaseOS/x86_64/os/repodata/repomd.xml 404
+30 mirror.admax.se http://mirror.admax.se/rocky/9.0/BaseOS/x86_64/os/repodata/repomd.xml 200
+195 mirror.uepg.br http://mirror.uepg.br/rocky/9.0/BaseOS/x86_64/os/repodata/repomd.xml 404
+247 mirrors.ipserverone.com http://mirrors.ipserverone.com/rocky/9.0/BaseOS/x86_64/os/repodata/repomd.xml 404'
+This SOP contains all the steps required by SIG/Core (a mix of Release +Engineering and Infrastructure) to perform releases of all Rocky Linux +versions. Work is in all collaboration within the entire group of +engineerings.
+| + | + |
|---|---|
| Owner | +SIG/Core (Release Engineering & Infrastructure) | +
| Email Contact | +infrastructure@rockylinux.org | +
| Email Contact | +releng@rockylinux.org | +
| Mattermost Contacts | +@label @neil @tgo @skip77 @mustafa @sherif @pgreco |
+
| Mattermost Channels | +~Infrastructure |
+
Within a minimum of two (2) days, the following should be true:
+Torrents should be setup. All files can be synced with the seed box(es) but +not yet published. The data should be verified using sha256sum and compared to +the CHECKSUM files provided with the files.
+Website should be ready (typically with an open PR in github). The content +should be verified that the design and content are correct and finalized.
+Enough mirrors should be setup. This essentially means that all content for +a release should be synced to our primary mirror with the executable bit turned +off, and the content should also be hard linked. In theory, mirror manager can +be queried to verify if mirrors are or appear to be in sync.
+Within a minimum of one (1) to two (2) days, the following should be true:
+Updates should be completed in the build system, and verified in staging.
+Updates should be sent to production and file lists updated to allow mirrors +to sync.
+Ensure the SIG/Core Checklist is read thoroughly and executed as listed.
+During release day, these should be verified/completed in order:
+Website - The primary website and user landing at rockylinux.org should allow +the user to efficiently click through to a download link of an ISO, image, or +torrent. It must be kept up.
+Torrent - The seed box(es) should be primed and ready to go for users +downloading via torrent.
+Release Notes & Documentation - The release notes are often on the same +website as the documentation. The main website and where applicable in the docs +should refer to the Release Notes of Rocky Linux.
+Wiki - If applicable, the necessary changes and resources should be available +for a release. In particular, if a major release has new repos, changed repo names, +this should be documented.
+Everything else!
+Mirror Manager
+Pass image to Testing Team for final validation
+Mirror Manager
+Pass image to Testing Team for validation
+Mirror Manager
+Pass image to Testing Team for final validation
+URL: https://accounts.rockylinux.org
+Purpose: Account Services maintains the accounts for almost all components of the Rocky ecosystem
+Technology: Noggin used by Fedora Infrastructure
+Contact: ~Infrastructure in Mattermost and #rockylinux-infra in Libera IRC
URL: https://git.resf.org
+Purpose: General projects, code, and so on for the Rocky Enterprise Software Foundation.
+Technology: Gitea
+Contact: ~Infrastructure, ~Development in Mattermost and #rockylinux-infra, #rockylinux-devel in Libera IRC
URL: https://github.com/rocky-linux
+Purpose: General purpose code, assets, and so on for Rocky Linux. Some content is mirrored to the RESF Git Service.
+Technology: GitHub
+Contact: ~Infrastructure, ~Development in Mattermost and #rockylinux-infra, #rockylinux-devel in Libera IRC
URL: https://git.rockylinux.org
+Purpose: Packages and light code for the Rocky Linux distribution
+Technology: GitLab
+Contact: ~Infrastructure, ~Development in Mattermost and #rockylinux-infra, #rockylinux-devel in Libera IRC
Purpose: Users can subscribe and interact with various mail lists for the Rocky ecosystem
+Technology: Mailman 3 + Hyper Kitty
+Contact: ~Infrastructure in Mattermost and #rockylinux-infra in Libera IRC
| Name | +Mattermost Name | +IRC Name | +|
|---|---|---|---|
| Louis Abel | +label@rockylinux.org | +@nazunalika | +Sokel/label/Sombra | +
| Mustafa Gezen | +mustafa@rockylinux.org | +@mustafa | +mstg | +
| Skip Grube | +skip@rockylinux.org | +@skip77 | ++ |
| Sherif Nagy | +sherif@rockylinux.org | +@sherif | ++ |
| Pablo Greco | +pgreco@rockylinux.org | +@pgreco | +pgreco | +
| Neil Hanlon | +neil@resf.org | +@neil | +neil | +
| Taylor Goodwill | +tg@resf.org | +@tgo | +tg | +
This SOP contains general checklists required by SIG/Core to prepare and plan +for the upcoming release. This work, in general, is required to be done on a +routine basis, even months out before the next major or minor release, as it +requires monitoring of upstream's (CentOS Stream) work to ensure Rocky Linux +will remain ready and compatible with Red Hat Enterprise Linux.
+| + | + |
|---|---|
| Owner | +SIG/Core (Release Engineering & Infrastructure) | +
| Email Contact | +infrastructure@rockylinux.org | +
| Email Contact | +releng@rockylinux.org | +
| Mattermost Contacts | +@label @neil @tgo @skip77 @mustafa @sherif @pgreco |
+
| Mattermost Channels | +~Infrastructure |
+
It is expected to monitor the following repositories upstream, as these will +indicate what is coming up for a given major or point release. These +repositories are found at the Red Hat gitlab.
+These repositories can be monitored by setting to "all activity" on the bell +icon.
+Upon changes to the upstream repositories, SIG/Core member should analyze the +changes and apply the same to the lookahead branches:
+rocky-release
+rocky-logos
+pungi-rocky
+sync-from-upstreamperidot-rocky
+comps
+sync-from-upstreamrocky-module-defaults
+sync-from-upstreamRepositories that generally track for LookAhead and Beta releases will flow +downward to the stable branch. For example:
+* rXs / rXlh
+ |
+ |----> rX-beta
+ |
+ |----> rX
+This applies to any specific rocky repo, such as comps, pungi, peridot-config, +and so on. As it is expected some repos will deviate in commit history, it is OK +to force push, under the assumption that changes made in the lower branch exists +in the upper branch. That way you can avoid changes/functionality being reverted +on accident.
+There are packages that are patched typically for the purpose of debranding. +List of patched packages are typically maintained in a metadata repository. The +obvious ones are listed below and should be monitored and maintained properly:
+