From 035a7b91ef3f317af94a89a01302dbedeb226da1 Mon Sep 17 00:00:00 2001 From: Louis Abel Date: Thu, 20 Oct 2022 01:14:06 -0700 Subject: [PATCH] Start adding documentation for empanadas --- docs/documentation/.pages | 1 + docs/documentation/empanadas.md | 57 ++++- docs/documentation/index.md | 2 +- docs/documentation/peridot.md | 7 + docs/documentation/references/.pages | 6 + .../references/empanadas_common.md | 215 ++++++++++++++++++ docs/documentation/references/index.md | 5 + docs/what_we_do.md | 3 + 8 files changed, 290 insertions(+), 6 deletions(-) create mode 100644 docs/documentation/peridot.md create mode 100644 docs/documentation/references/.pages create mode 100644 docs/documentation/references/empanadas_common.md create mode 100644 docs/documentation/references/index.md create mode 100644 docs/what_we_do.md diff --git a/docs/documentation/.pages b/docs/documentation/.pages index c0a12ff..eb7b5b2 100644 --- a/docs/documentation/.pages +++ b/docs/documentation/.pages @@ -3,4 +3,5 @@ nav: - ... | index.md - Composing Releases: compose.md - Empanadas: empanadas.md + - References: references ... diff --git a/docs/documentation/empanadas.md b/docs/documentation/empanadas.md index 2f71259..f8765b0 100644 --- a/docs/documentation/empanadas.md +++ b/docs/documentation/empanadas.md @@ -22,22 +22,69 @@ debugging of repositories and its metadata. with the goal to try to emulate the Fedora Project's pungi to an extent. While it is not perfect, it achieves the very basic goals of creating repositories, images and ISO's for consumption by the end user. It also has interactions with peridot, -the build system used by Rocky Linux to build the distribution. +the build system used by the RESF to build the Rocky Linux distribution. For performing syncs, it relies on the use of podman to perform syncing in a parallel fashion. This was done because it is not possible to run multiple dnf transactions at once on a single system and looping one repository at a time is -not sustainable. +not sustainable (nor fast). ## Requirements -* Poetry must be available on the system +* Poetry must be installed on the system * Podman must be installed on the system -* fpart must be installed on the system (available in EPEL) +* `fpart` must be installed on the system (available in EPEL on EL systems) * Enough storage should be available if repositories are being synced +* `mock` must be installed if building live images +* System must be an Enterprise Linux system or Fedora with the `%rhel` macro set + +## Features + +As of this writing, `empanadas` has the following abilities: + +* Repository syncing via dnf from a peridot instance or applicable repos +* Per profile dnf repoclosure checking for all applicable repos +* Per profile dnf repoclosure checking for peridot instance repositories +* Basic ISO Building via `lorax` +* Extra ISO Building via `xorriso` for DVD and minimal images +* Live ISO Building using `livemedia-creator` and `mock` +* Anaconda treeinfo builder +* Cloud Image builder + +## Installing Empanadas + +The below is how to install empanadas from the development branch on a Fedora +system. + +``` +% dnf install git podman fpart poetry mock -y +% git clone https://git.resf.org/sig_core/toolkit.git -b devel +% cd toolkit/iso/empanadas +% poetry install +``` + +## Configuring Empanadas + +Depending on how you are using empanadas will depend on how your configurations +will be setup. + +* `empanadas/common.py` +* `empanadas/config/*.yaml` +* `empanadas/sig/*.yaml` + +These configuration files are delicate and can control a wide variety of the +moving parts of empanadas. As these configurations are fairly massive, we +recommend checking the [reference guides](./references/) for deeper details into +configuring for base distribution or "SIG" content. ## Using Empanadas -### +The most common way to use empanadas is to sync repositories from a peridot +instance. This is performed upon each release or on each set of updates as they +come from upstream. Below lists how to use `empanadas`, as well as the common +options. + +Note that for each of these commands, it is fully expected you are running +`poetry run` in the root of empanadas. {% include "resources_bottom.md" %} diff --git a/docs/documentation/index.md b/docs/documentation/index.md index 14d4ffc..3d6ed65 100644 --- a/docs/documentation/index.md +++ b/docs/documentation/index.md @@ -24,7 +24,7 @@ The tools in use for the distribution are in the table below. For Rocky Linux to be build, we use `Peridot` as the build system and `empanadas` to "compose" the distribution. As we do not use Koji for Rocky -Linux beyond version 9, pungi no longer can be used. Peridot instead takes +Linux beyond version 9, pungi can no longer be used. Peridot instead takes pungi configuration data and comps and transforms them into a format it can understand. Empanadas then comes in to do the "compose" and sync all the repositories down. diff --git a/docs/documentation/peridot.md b/docs/documentation/peridot.md new file mode 100644 index 0000000..af33340 --- /dev/null +++ b/docs/documentation/peridot.md @@ -0,0 +1,7 @@ +--- +title: Peridot Build System +--- + +This page goes over the Peridot Build System and how SIG/Core utilizes it. + +More to come. diff --git a/docs/documentation/references/.pages b/docs/documentation/references/.pages new file mode 100644 index 0000000..7c432e2 --- /dev/null +++ b/docs/documentation/references/.pages @@ -0,0 +1,6 @@ +--- +nav: + - ... | index.md + - ... | empanadas_common.md + - ... | empanadas_config.md + - ... | empanadas_sig_config.md diff --git a/docs/documentation/references/empanadas_common.md b/docs/documentation/references/empanadas_common.md new file mode 100644 index 0000000..9fc255f --- /dev/null +++ b/docs/documentation/references/empanadas_common.md @@ -0,0 +1,215 @@ +--- +title: Empanadas common.py Configuration +--- + +## Config Items + +### config.rlmacro + +type: String + +required: True + +description: Empanadas expects to run on an EL system. This is part of the +general check up. It should not be hardcoded and use the rpm python module. + +### config.dist + +type: String + +required: False + +description: Was the original tag placed in mock configs. This combines `el` +with the rpm python module expansion. This is no longer required. The +option is still available for future use. + +### config.arch + +type: String + +required: True + +description: The architecture of the current running system. This is checked +against the supported architectures in general release configurations. This +should not be hardcoded. + +### config.date_stamp + +type: String + +required: True + +description: Date time stamp in the form of YYYYMMDD.HHMMSS. This should not be +hardcoded. + +### config.compose_root + +type: String + +required: True + +description: Root path of composes on the system running empanadas. + +### config.staging_root + +type: String + +required: False + +description: For future use. Root path of staging repository location where +content will be synced to. + +### config.production_root + +type: String + +required: False + +description: For future use. Root path of production repository location where +content will be synced to from staging. + +### config.category_stub + +type: String + +required: True + +description: For future use. Stub path that is appended to `staging_root` and +`production_root`. + +example: `mirror/pub/rocky` + +### config.sig_category_stub + +type: String + +required: True + +description: For future use. Stub path that is appended to `staging_root` and +`production_root` for SIG content. + +example: `mirror/pub/sig` + +### config.repo_base_url + +type: String + +required: True + +description: URL to the base url's where the repositories live. This is +typically to a peridot instance. This is supplemented by the configuration +`project_id` parameter. + +Note that this does not have to be a peridot instance. The combination of this +value and `project_id` can be sufficient enough for empanadas to perform its +work. + +### config.mock_work_root + +type: String + +required: True + +description: Hardcoded path to where ISO work is performed within a mock chroot. +This is the default path created by mock and it is recommended not to change +this. + +example: `/builddir` + +### config.container + +type: String + +required: True + +description: This is the container used to perform all operations in podman. + +example: `centos:stream9` + +### config.distname + +type: String + +required: True + +description: Name of the distribution you are building or building for. + +example: `Rocky Linux` + +### config.shortname + +type: String + +required: True + +description: Short name of the distribution you are building or building for. + +example: `Rocky` + +### config.translators + +type: Dictionary + +required: True + +description: Translates Linux architectures to golang architectures. Reserved +for future use. + +### config.aws_region + +type: String + +required: False + +description: Region you are working in with AWS or onprem cloud that supports +this variable. + +example: `us-east-2` + +### config.bucket + +type: String + +required: False + +description: Name of the S3-compatible bucket that is used to pull images from. +Requires `aws_region`. + +### config.bucket_url + +type: String + +required: False + +description: URL of the S3-compatible bucket that is used to pull images from. + +## Reference Example + +``` +config = { + "rlmacro": rpm.expandMacro('%rhel'), + "dist": 'el' + rpm.expandMacro('%rhel'), + "arch": platform.machine(), + "date_stamp": time.strftime("%Y%m%d.%H%M%S", time.localtime()), + "compose_root": "/mnt/compose", + "staging_root": "/mnt/repos-staging", + "production_root": "/mnt/repos-production", + "category_stub": "mirror/pub/rocky", + "sig_category_stub": "mirror/pub/sig", + "repo_base_url": "https://yumrepofs.build.resf.org/v1/projects", + "mock_work_root": "/builddir", + "container": "centos:stream9", + "distname": "Rocky Linux", + "shortname": "Rocky", + "translators": { + "x86_64": "amd64", + "aarch64": "arm64", + "ppc64le": "ppc64le", + "s390x": "s390x", + "i686": "386" + }, + "aws_region": "us-east-2", + "bucket": "resf-empanadas", + "bucket_url": "https://resf-empanadas.s3.us-east-2.amazonaws.com" +} +``` diff --git a/docs/documentation/references/index.md b/docs/documentation/references/index.md new file mode 100644 index 0000000..9a03678 --- /dev/null +++ b/docs/documentation/references/index.md @@ -0,0 +1,5 @@ +--- +title: References +--- + +Use this section to locate reference configuration items for the toolkit. diff --git a/docs/what_we_do.md b/docs/what_we_do.md new file mode 100644 index 0000000..a73473b --- /dev/null +++ b/docs/what_we_do.md @@ -0,0 +1,3 @@ +--- +title: What We Do +---