Merge pull request #16 from dirkmueller/openela

Support the openELA "PATCHES" directory format as well
2024-06-01 16:00:14 +00:00 · 2023-11-16 18:04:50 +01:00 · 2023-11-06 16:53:43 +01:00 · 2023-10-06 08:13:19 +02:00 · 2023-10-06 08:05:29 +02:00 · 2023-10-06 07:56:14 +02:00
26 changed files with 1573 additions and 354 deletions
--- a/.github/workflows/go.yml
+++ b/.github/workflows/go.yml
@ -16,7 +16,7 @@ jobs:
      - name: Set up Go
        uses: actions/setup-go@v2
        with:
-          go-version: 1.16
+          go-version: 1.18
      - name: Run GoReleaser
        uses: goreleaser/goreleaser-action@v2
        with:
--- a/.goreleaser.yml
+++ b/.goreleaser.yml
@ -14,12 +14,16 @@ builds:
    goarch:
      - amd64
      - arm64
+      - s390x
+      - ppc64le
 archives:
-  - replacements:
-      darwin: Darwin
-      linux: Linux
-      amd64: x86_64
-      arm64: aarch64
+  - name_template: >-
+      {{- .ProjectName }}_
+      {{- .Version }}
+      {{- title .Os }}_
+      {{- if eq .Arch "amd64" }}x86_64
+      {{- else if eq .Arch "arm64" }}aarch64
+      {{- else }}{{ .Arch }}{{ end }}{{ end -}}
 checksum:
  name_template: 'checksums.txt'
 snapshot:
--- a/README.md
+++ b/README.md
@ -1,19 +1,23 @@
 # srpmproc
 Upstream package importer with auto patching. Reference implementation for OpenPatch

-# Usage
+## Usage
 ```
 Usage:
  srpmproc [flags]
  srpmproc [command]

 Available Commands:
-  fetch
+  fetch       
  help        Help about any command

 Flags:
-      --allow-stream-branches           Allow import from stream branches
+      --basic-password string           Basic auth password
+      --basic-username string           Basic auth username
      --branch-prefix string            Branch prefix (replaces import-branch-prefix) (default "r")
+      --branch-suffix string            Branch suffix to use for imported branches
+      --cdn string                      CDN URL shortcuts for well-known distros, auto-assigns --cdn-url.  Valid values:  rocky8, rocky, fedora, centos, centos-stream.  Setting this overrides --cdn-url
+      --cdn-url string                  CDN URL to download blobs from. Simple URL follows default rocky/centos patterns. Can be customized using macros (see docs) (default "https://git.centos.org/sources")
      --git-committer-email string      Email of committer (default "rockyautomation@rockylinux.org")
      --git-committer-name string       Name of committer (default "rockyautomation")
  -h, --help                            help for srpmproc
@ -25,16 +29,51 @@ Flags:
      --no-dup-mode                     If enabled, skips already imported tags
      --no-storage-download             If enabled, blobs are always downloaded from upstream
      --no-storage-upload               If enabled, blobs are not uploaded to blob storage
+      --package-release string          Package release to fetch
+      --package-version string          Package version to fetch
      --rpm-prefix string               Where to retrieve SRPM content. Only used when source-rpm is not a local file (default "https://git.centos.org/rpms")
      --single-tag string               If set, only this tag is imported
      --source-rpm string               Location of RPM to process
      --ssh-key-location string         Location of the SSH key to use to authenticate against upstream
      --ssh-user string                 SSH User (default "git")
      --storage-addr string             Bucket to use as blob storage
+      --strict-branch-mode              If enabled, only branches with the calculated name are imported and not prefix only
+      --taglessmode                     Tagless mode:  If set, pull the latest commit from the branch and determine version numbers from spec file.  This is auto-tried if tags aren't found.
      --tmpfs-mode string               If set, packages are imported to path and patched but not pushed
      --upstream-prefix string          Upstream git repository prefix
-      --upstream-prefix-https string    Web version of upstream prefix. Required if module-mode
      --version int                     Upstream version

 Use "srpmproc [command] --help" for more information about a command.
 ```
+
+<br />
+
+## Examples:
+
+1. Import the kernel package from git.centos.org/rpms/, to local folder /opt/gitroot/rpms/kernel.git/ .  Download the lookaside source tarballs from the default CentOS file server location to local folder `/opt/fake_s3/` .  We want to grab branch "c8" (import prefix plus RHEL version), and it will be committed as branch "r8" (branch prefix plus RHEL version).  This assumes that `/opt/fake_s3` exists, and `/opt/gitroot/rpms/kernel.git` exists and is a git repository of some kind (even an empty one).
+
+```
+srpmproc --branch-prefix "r"  --import-branch-prefix "c"  --rpm-prefix "https://git.centos.org/rpms" --version 8 --storage-addr file:///opt/fake_s3  --upstream-prefix file:///opt/gitroot   --cdn centos --strict-branch-mode --source-rpm kernel
+```
+
+<br />
+
+## CDN and --cdn-url
+The --cdn-url option allows for Go-style templates to craft complex URL patterns.  These templates are: `{{.Name}}` (package name), `{{.Hash}}` (hash of lookaside file), `{{.Hashtype}}` (hash type of file, like "sha256" or "sha512"), `{{.Branch}}` (the branch we are importing), and `{{.Filename}}` (the lookaside file's name as it appears in SOURCES/).  You can add these values as part of --cdn-url to craft your lookaside pattern.
+
+
+For example, if I wanted my lookaside downloads to come from CentOS 9 Stream, I would use as part of my command:
+```
+--cdn-url "https://sources.stream.centos.org/sources/rpms/{{.Name}}/{{.Filename}}/{{.Hashtype}}/{{.Hash}}/{{.Filename}}"
+```
+
+
+**Default Behavior:**  If these templates are not used, the default behavior of `--cdn-url` is to fall back on the traditional RHEL import pattern:  `<CDN_URL>/<NAME>/<BRANCH>/<HASH>` .  If that fails, a further fallback is attempted, the simple: `<CDN_URL>/<HASH>`.  These cover the common Rocky Linux and RHEL/CentOS imports if the base lookaside URL is the only thing given.  If no `--cdn-url` is specified, it defaults to "https://git.centos.org/sources" (for RHEL imports into Rocky Linux)
+
+
+**CDN Shorthand:** For convenience, some lookaside patterns for popular distros are provided via the `--cdn` option.  You can specify this without needing to use the longer `--cdn-url`.  For example, when importing from CentOS 9 Stream, you could use `--cdn centos-stream`
+
+
+
+
+
--- a/cmd/srpmproc/fetch.go
+++ b/cmd/srpmproc/fetch.go
@ -21,10 +21,12 @@
 package main

 import (
-	"github.com/rocky-linux/srpmproc/pkg/srpmproc"
-	"github.com/spf13/cobra"
 	"log"
 	"os"
+
+	"github.com/go-git/go-billy/v5/osfs"
+	"github.com/rocky-linux/srpmproc/pkg/srpmproc"
+	"github.com/spf13/cobra"
 )

 var fetch = &cobra.Command{
@ -37,6 +39,8 @@ var cdnUrl string
 func init() {
 	fetch.Flags().StringVar(&cdnUrl, "cdn-url", "", "Path to CDN")
 	_ = fetch.MarkFlagRequired("cdn-url")
+
+	root.AddCommand(fetch)
 }

 func runFetch(_ *cobra.Command, _ []string) {
@ -45,12 +49,8 @@ func runFetch(_ *cobra.Command, _ []string) {
 		log.Fatalf("could not get working directory: %v", err)
 	}

-	err = srpmproc.Fetch(os.Stdout, cdnUrl, wd)
+	err = srpmproc.Fetch(os.Stdout, cdnUrl, wd, osfs.New("/"), nil)
 	if err != nil {
 		log.Fatal(err)
 	}
 }
-
-func init() {
-	root.AddCommand(fetch)
-}
--- a/cmd/srpmproc/main.go
+++ b/cmd/srpmproc/main.go
@ -22,10 +22,11 @@ package main

 import (
 	"encoding/json"
-	"github.com/rocky-linux/srpmproc/pkg/srpmproc"
 	"log"
 	"os"

+	"github.com/rocky-linux/srpmproc/pkg/srpmproc"
+
 	"github.com/spf13/cobra"
 )

@ -50,9 +51,15 @@ var (
 	noStorageUpload      bool
 	manualCommits        string
 	moduleFallbackStream string
-	allowStreamBranches  bool
+	branchSuffix         string
+	strictBranchMode     bool
 	basicUsername        string
 	basicPassword        string
+	packageVersion       string
+	packageRelease       string
+	taglessMode          bool
+	cdn                  string
+	moduleBranchNames    bool
 )

 var root = &cobra.Command{
@ -78,7 +85,8 @@ func mn(_ *cobra.Command, _ []string) {
 		ImportBranchPrefix:   importBranchPrefix,
 		BranchPrefix:         branchPrefix,
 		NoDupMode:            noDupMode,
-		AllowStreamBranches:  allowStreamBranches,
+		BranchSuffix:         branchSuffix,
+		StrictBranchMode:     strictBranchMode,
 		ModuleFallbackStream: moduleFallbackStream,
 		NoStorageUpload:      noStorageUpload,
 		NoStorageDownload:    noStorageDownload,
@ -86,6 +94,11 @@ func mn(_ *cobra.Command, _ []string) {
 		CdnUrl:               cdnUrl,
 		HttpUsername:         basicUsername,
 		HttpPassword:         basicPassword,
+		PackageVersion:       packageVersion,
+		PackageRelease:       packageRelease,
+		TaglessMode:          taglessMode,
+		Cdn:                  cdn,
+		ModuleBranchNames:    moduleBranchNames,
 	})
 	if err != nil {
 		log.Fatal(err)
@ -100,7 +113,6 @@ func mn(_ *cobra.Command, _ []string) {
 	if err != nil {
 		log.Fatal(err)
 	}
-
 }

 func main() {
@ -121,7 +133,7 @@ func main() {
 	root.Flags().StringVar(&rpmPrefix, "rpm-prefix", "https://git.centos.org/rpms", "Where to retrieve SRPM content. Only used when source-rpm is not a local file")
 	root.Flags().StringVar(&importBranchPrefix, "import-branch-prefix", "c", "Import branch prefix")
 	root.Flags().StringVar(&branchPrefix, "branch-prefix", "r", "Branch prefix (replaces import-branch-prefix)")
-	root.Flags().StringVar(&cdnUrl, "cdn-url", "https://git.centos.org/sources", "CDN URL to download blobs from")
+	root.Flags().StringVar(&cdnUrl, "cdn-url", "https://git.centos.org/sources", "CDN URL to download blobs from. Simple URL follows default rocky/centos patterns. Can be customized using macros (see docs)")
 	root.Flags().StringVar(&singleTag, "single-tag", "", "If set, only this tag is imported")
 	root.Flags().BoolVar(&noDupMode, "no-dup-mode", false, "If enabled, skips already imported tags")
 	root.Flags().BoolVar(&moduleMode, "module-mode", false, "If enabled, imports a module instead of a package")
@ -130,9 +142,15 @@ func main() {
 	root.Flags().BoolVar(&noStorageUpload, "no-storage-upload", false, "If enabled, blobs are not uploaded to blob storage")
 	root.Flags().StringVar(&manualCommits, "manual-commits", "", "Comma separated branch and commit list for packages with broken release tags (Format: BRANCH:HASH)")
 	root.Flags().StringVar(&moduleFallbackStream, "module-fallback-stream", "", "Override fallback stream. Some module packages are published as collections and mostly use the same stream name, some of them deviate from the main stream")
-	root.Flags().BoolVar(&allowStreamBranches, "allow-stream-branches", false, "Allow import from stream branches")
+	root.Flags().StringVar(&branchSuffix, "branch-suffix", "", "Branch suffix to use for imported branches")
+	root.Flags().BoolVar(&strictBranchMode, "strict-branch-mode", false, "If enabled, only branches with the calculated name are imported and not prefix only")
 	root.Flags().StringVar(&basicUsername, "basic-username", "", "Basic auth username")
 	root.Flags().StringVar(&basicPassword, "basic-password", "", "Basic auth password")
+	root.Flags().StringVar(&packageVersion, "package-version", "", "Package version to fetch")
+	root.Flags().StringVar(&packageRelease, "package-release", "", "Package release to fetch")
+	root.Flags().BoolVar(&taglessMode, "taglessmode", false, "Tagless mode:  If set, pull the latest commit from the branch and determine version numbers from spec file.  This is auto-tried if tags aren't found.")
+	root.Flags().StringVar(&cdn, "cdn", "", "CDN URL shortcuts for well-known distros, auto-assigns --cdn-url.  Valid values:  rocky8, rocky, fedora, centos, centos-stream.  Setting this overrides --cdn-url")
+	root.Flags().BoolVar(&moduleBranchNames, "module-branch-names-only", false, "If enabled, module imports will use the branch name that is being imported, rather than use the commit hash.")

 	if err := root.Execute(); err != nil {
 		log.Fatal(err)
--- a/go.mod
+++ b/go.mod
@ -1,18 +1,60 @@
 module github.com/rocky-linux/srpmproc

-go 1.15
+go 1.18

 require (
 	cloud.google.com/go/storage v1.12.0
 	github.com/aws/aws-sdk-go v1.36.12
 	github.com/bluekeyes/go-gitdiff v0.5.0
-	github.com/cavaliercoder/go-cpio v0.0.0-20180626203310-925f9528c45e
-	github.com/cavaliercoder/go-rpm v0.0.0-20200122174316-8cb9fd9c31a8
 	github.com/go-git/go-billy/v5 v5.0.0
 	github.com/go-git/go-git/v5 v5.2.0
-	github.com/golang/protobuf v1.4.2
 	github.com/spf13/cobra v1.1.1
 	github.com/spf13/viper v1.7.0
 	google.golang.org/protobuf v1.25.0
-	gopkg.in/yaml.v2 v2.2.8
+	gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b
+)
+
+require (
+	cloud.google.com/go v0.66.0 // indirect
+	github.com/emirpasic/gods v1.12.0 // indirect
+	github.com/fsnotify/fsnotify v1.4.7 // indirect
+	github.com/go-git/gcfg v1.5.0 // indirect
+	github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e // indirect
+	github.com/golang/protobuf v1.4.2 // indirect
+	github.com/googleapis/gax-go/v2 v2.0.5 // indirect
+	github.com/hashicorp/hcl v1.0.0 // indirect
+	github.com/imdario/mergo v0.3.9 // indirect
+	github.com/inconshreveable/mousetrap v1.0.0 // indirect
+	github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 // indirect
+	github.com/jmespath/go-jmespath v0.4.0 // indirect
+	github.com/jstemmer/go-junit-report v0.9.1 // indirect
+	github.com/kevinburke/ssh_config v0.0.0-20190725054713-01f96b0aa0cd // indirect
+	github.com/magiconair/properties v1.8.1 // indirect
+	github.com/mitchellh/go-homedir v1.1.0 // indirect
+	github.com/mitchellh/mapstructure v1.1.2 // indirect
+	github.com/pelletier/go-toml v1.2.0 // indirect
+	github.com/sergi/go-diff v1.1.0 // indirect
+	github.com/spf13/afero v1.1.2 // indirect
+	github.com/spf13/cast v1.3.0 // indirect
+	github.com/spf13/jwalterweatherman v1.0.0 // indirect
+	github.com/spf13/pflag v1.0.5 // indirect
+	github.com/subosito/gotenv v1.2.0 // indirect
+	github.com/xanzy/ssh-agent v0.2.1 // indirect
+	go.opencensus.io v0.22.4 // indirect
+	golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9 // indirect
+	golang.org/x/lint v0.0.0-20200302205851-738671d3881b // indirect
+	golang.org/x/mod v0.3.0 // indirect
+	golang.org/x/net v0.0.0-20201110031124-69a78807bb2b // indirect
+	golang.org/x/oauth2 v0.0.0-20200902213428-5d25da1a8d43 // indirect
+	golang.org/x/sys v0.1.0 // indirect
+	golang.org/x/text v0.3.3 // indirect
+	golang.org/x/tools v0.0.0-20200918232735-d647fc253266 // indirect
+	golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect
+	google.golang.org/api v0.32.0 // indirect
+	google.golang.org/appengine v1.6.6 // indirect
+	google.golang.org/genproto v0.0.0-20200921151605-7abf4a1a14d5 // indirect
+	google.golang.org/grpc v1.32.0 // indirect
+	gopkg.in/ini.v1 v1.51.0 // indirect
+	gopkg.in/warnings.v0 v0.1.2 // indirect
+	gopkg.in/yaml.v2 v2.2.8 // indirect
 )
--- a/go.sum
+++ b/go.sum
@ -36,18 +36,20 @@ cloud.google.com/go/storage v1.10.0/go.mod h1:FLPqc6j+Ki4BU591ie1oL6qBQGu2Bl/tZ9
 cloud.google.com/go/storage v1.12.0 h1:4y3gHptW1EHVtcPAVE0eBBlFuGqEejTTG3KdIE0lUX4=
 cloud.google.com/go/storage v1.12.0/go.mod h1:fFLk2dp2oAhDz8QFKwqrjdJvxSp/W2g7nillojlL5Ho=
 dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
+github.com/BurntSushi/toml v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=
 github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
 github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=
 github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU=
+github.com/alcortesm/tgz v0.0.0-20161220082320-9c5fe88206d7 h1:uSoVVbwJiQipAclBbw+8quDsfcvFjOpI5iCf4p/cqCs=
 github.com/alcortesm/tgz v0.0.0-20161220082320-9c5fe88206d7/go.mod h1:6zEj6s6u/ghQa61ZWa/C2Aw3RkjiTBOix7dkqa1VLIs=
-github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc h1:cAKDfWh5VpdgMhJosfJnn5/FoN2SRZ4p7fJNX58YPaU=
 github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
-github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf h1:qet1QNfXsQxTZqLG4oE62mJzwPIB8+Tee4RNCL9ulrY=
 github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
+github.com/anmitsu/go-shlex v0.0.0-20161002113705-648efa622239 h1:kFOfPq6dUM1hTo4JG6LR5AXSUEsOjtdm0kw0FtQtMJA=
 github.com/anmitsu/go-shlex v0.0.0-20161002113705-648efa622239/go.mod h1:2FmKhYUyUczH0OGQWaF5ceTx0UBShxjsH6f8oGKYe2c=
 github.com/armon/circbuf v0.0.0-20150827004946-bbbad097214e/go.mod h1:3U/XgcO3hCbHZ8TKRvWD2dDTCfh9M9ya+I9JpbB7O8o=
 github.com/armon/go-metrics v0.0.0-20180917152333-f0300d1749da/go.mod h1:Q73ZrmVTwzkszR9V5SSuryQ31EELlFMUz1kKyl939pY=
 github.com/armon/go-radix v0.0.0-20180808171621-7fddfc383310/go.mod h1:ufUuZ+zHj4x4TnLV4JWEpy2hxWSpsRywHrMgIH9cCH8=
+github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5 h1:0CwZNZbxp69SHPdPJAN/hZIm0C4OItdklCFmMRWYpio=
 github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5/go.mod h1:wHh0iHkYZB8zMSxRWpUBQtwG5a7fFgvEO+odwuTv2gs=
 github.com/aws/aws-sdk-go v1.36.12 h1:YJpKFEMbqEoo+incs5qMe61n1JH3o4O1IMkMexLzJG8=
 github.com/aws/aws-sdk-go v1.36.12/go.mod h1:hcU610XS61/+aQV88ixoOzUoG7v3b31pl2zKMmprdro=
@ -57,10 +59,6 @@ github.com/bgentry/speakeasy v0.1.0/go.mod h1:+zsyZBPWlz7T6j88CTgSN5bM796AkVf0kB
 github.com/bketelsen/crypt v0.0.3-0.20200106085610-5cbc8cc4026c/go.mod h1:MKsuJmJgSg28kpZDP6UIiPt0e0Oz0kqKNGyRaWEPv84=
 github.com/bluekeyes/go-gitdiff v0.5.0 h1:AXrIoy/VEA9Baz2lhwMlpdzDJ/sKof6C9yTt1oqw4hQ=
 github.com/bluekeyes/go-gitdiff v0.5.0/go.mod h1:QpfYYO1E0fTVHVZAZKiRjtSGY9823iCdvGXBcEzHGbM=
-github.com/cavaliercoder/go-cpio v0.0.0-20180626203310-925f9528c45e h1:hHg27A0RSSp2Om9lubZpiMgVbvn39bsUmW9U5h0twqc=
-github.com/cavaliercoder/go-cpio v0.0.0-20180626203310-925f9528c45e/go.mod h1:oDpT4efm8tSYHXV5tHSdRvBet/b/QzxZ+XyyPehvm3A=
-github.com/cavaliercoder/go-rpm v0.0.0-20200122174316-8cb9fd9c31a8 h1:jP7ki8Tzx9ThnFPLDhBYAhEpI2+jOURnHQNURgsMvnY=
-github.com/cavaliercoder/go-rpm v0.0.0-20200122174316-8cb9fd9c31a8/go.mod h1:AZIh1CCnMrcVm6afFf96PBvE2MRpWFco91z8ObJtgDY=
 github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
 github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc=
 github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI=
@ -76,6 +74,7 @@ github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f/go.mod h1:E3G3o1h8I7cfc
 github.com/cpuguy83/go-md2man/v2 v2.0.0/go.mod h1:maD7wRr/U5Z6m/iR4s+kqSMx2CaBsrgA7czyZG/E6dU=
 github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZmtrrCbhqsmaPHjLKYnJCaQ=
 github.com/dgryski/go-sip13 v0.0.0-20181026042036-e10d5fee7954/go.mod h1:vAd38F8PWV+bWy6jNmig1y/TA+kYO4g3RSRF0IAv0no=
@ -90,13 +89,13 @@ github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568/go.mod h1:xEzjJPgXI
 github.com/fsnotify/fsnotify v1.4.7 h1:IXs+QLmnXW2CcXuY+8Mzv/fWEsPGWxqefPtCP5CnV9I=
 github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
 github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
+github.com/gliderlabs/ssh v0.2.2 h1:6zsha5zo/TWhRhwqCD3+EarCAgZ2yN28ipRnGPnwkI0=
 github.com/gliderlabs/ssh v0.2.2/go.mod h1:U7qILu1NlMHj9FlMhZLlkCdDnU1DBEAqr0aevW3Awn0=
 github.com/go-git/gcfg v1.5.0 h1:Q5ViNfGF8zFgyJWPqYwA7qGFoMTEiBmdlkcfRmpIMa4=
 github.com/go-git/gcfg v1.5.0/go.mod h1:5m20vg6GwYabIxaOonVkTdrILxQMpEShl1xiMF4ua+E=
 github.com/go-git/go-billy/v5 v5.0.0 h1:7NQHvd9FVid8VL4qVUMm8XifBK+2xCoZ2lSk0agRrHM=
 github.com/go-git/go-billy/v5 v5.0.0/go.mod h1:pmpqyWchKfYfrkb/UVH4otLvyi/5gJlGI4Hb3ZqZ3W0=
-github.com/go-git/go-git v1.0.0 h1:YcN9iDGDoXuIw0vHls6rINwV416HYa0EB2X+RBsyYp4=
-github.com/go-git/go-git v4.7.0+incompatible h1:+W9rgGY4DOKKdX2x6HxSR7HNeTxqiKrOvKnuittYVdA=
+github.com/go-git/go-git-fixtures/v4 v4.0.2-0.20200613231340-f56387b50c12 h1:PbKy9zOy4aAKrJ5pibIRpVO2BXnK1Tlcg+caKI7Ox5M=
 github.com/go-git/go-git-fixtures/v4 v4.0.2-0.20200613231340-f56387b50c12/go.mod h1:m+ICp2rF3jDhFgEZ/8yziagdT1C+ZpZcrJjappBCDSw=
 github.com/go-git/go-git/v5 v5.2.0 h1:YPBLG/3UK1we1ohRkncLjaXWLW+HKp5QNM/jTli2JgI=
 github.com/go-git/go-git/v5 v5.2.0/go.mod h1:kh02eMX+wdqqxgNMEyq8YgwlIOsDOa9homkUq1PoTMs=
@ -145,8 +144,11 @@ github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/
 github.com/google/go-cmp v0.4.1/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
 github.com/google/go-cmp v0.5.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
 github.com/google/go-cmp v0.5.1/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
+github.com/google/go-cmp v0.5.2 h1:X2ev0eStA3AbceY54o37/0PQ/UWqKEiiO2dKL5OPaFM=
 github.com/google/go-cmp v0.5.2/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
+github.com/google/martian v2.1.0+incompatible h1:/CP5g8u/VJHijgedC/Legn3BAbAaWPgecwXBIDzw5no=
 github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=
+github.com/google/martian/v3 v3.0.0 h1:pMen7vLs8nvgEYhywH3KDWJIJTeEr2ULsVWHWYHQyBs=
 github.com/google/martian/v3 v3.0.0/go.mod h1:y5Zk1BBys9G+gd6Jrk0W3cC1+ELVxBWuIGO+w/tUAp0=
 github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
 github.com/google/pprof v0.0.0-20190515194954-54271f7e092f/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
@ -160,6 +162,7 @@ github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm4
 github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg=
 github.com/googleapis/gax-go/v2 v2.0.5 h1:sjZBwGj9Jlw33ImPtvFviGYvseOtDM7hkSKB7+Tv3SM=
 github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk=
+github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1 h1:EGx4pi6eqNxGaHF6qqu48+N2wcFQ5qg5FXgOdqsJ5d8=
 github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY=
 github.com/gorilla/websocket v1.4.2/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
 github.com/grpc-ecosystem/go-grpc-middleware v1.0.0/go.mod h1:FiyG127CGDf3tlThmgyCl78X/SZQqEOJBCDaAfeWzPs=
@ -196,12 +199,14 @@ github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99/go.mod h1:1lJo3i
 github.com/jessevdk/go-flags v1.4.0/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI=
 github.com/jmespath/go-jmespath v0.4.0 h1:BEgLn5cpjn8UN1mAw4NjwDrS35OdebyEtFe+9YPoQUg=
 github.com/jmespath/go-jmespath v0.4.0/go.mod h1:T8mJZnbsbmF+m6zOOFylbeCJqk5+pHWvzYPziyZiYoo=
+github.com/jmespath/go-jmespath/internal/testify v1.5.1 h1:shLQSRRSCCPj3f2gpwzGwWFoC7ycTf1rcQZHOlsJ6N8=
 github.com/jmespath/go-jmespath/internal/testify v1.5.1/go.mod h1:L3OGu8Wl2/fWfCI6z80xFu9LTZmf1ZRjMHUOPmWr69U=
 github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo=
 github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU=
 github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU=
 github.com/jstemmer/go-junit-report v0.9.1 h1:6QPYqodiu3GuPL+7mfx+NwDdp2eTkp9IfEUpgAwUN0o=
 github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/XSXhF0NWZEnDohbsk=
+github.com/jtolds/gls v4.20.0+incompatible h1:xdiiI2gbIgH/gLH7ADydsJ1uDOEzR8yvV7C0MuV77Wo=
 github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU=
 github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w=
 github.com/kevinburke/ssh_config v0.0.0-20190725054713-01f96b0aa0cd h1:Coekwdh0v2wtGp9Gmz1Ze3eVRAWJMLokvN3QjdzCHLY=
@ -213,6 +218,7 @@ github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFB
 github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
 github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
 github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
+github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
 github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
 github.com/magiconair/properties v1.8.1 h1:ZC2Vc7/ZFkGmsVC9KvOjumD+G5lXy2RtTKyzRKO2BQ4=
 github.com/magiconair/properties v1.8.1/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
@ -233,6 +239,7 @@ github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh
 github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
 github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
 github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
+github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e h1:fD57ERR4JtEqsWbfPhv4DMiApHyliiK5xCTNVSPiaAs=
 github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e/go.mod h1:zD1mROLANZcx1PVRCS0qkT7pwLkGfwJo4zjcN/Tysno=
 github.com/oklog/ulid v1.3.1/go.mod h1:CirwcVhetQ6Lv90oh/F+FBtV6XMibvdAFo93nm5qn4U=
 github.com/pascaldekloe/goe v0.0.0-20180627143212-57f6aae5913c/go.mod h1:lzWF7FIEvWOWxwDKqyGYQf6ZUaNfKdP144TG7ZOy1lc=
@ -240,7 +247,9 @@ github.com/pelletier/go-toml v1.2.0 h1:T5zMGML61Wp+FlcbWjRDT7yAxhJNAiPPLOFECq181
 github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic=
 github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
 github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
+github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
 github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
+github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
 github.com/posener/complete v1.1.1/go.mod h1:em0nMJCgc9GFtwrmVmEMR/ZL6WyhyjMBndrE9hABlRI=
 github.com/prometheus/client_golang v0.9.1/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw=
@ -249,7 +258,6 @@ github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:
 github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
 github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
 github.com/prometheus/common v0.0.0-20181113130724-41aa239b4cce/go.mod h1:daVV7qP5qjZbuso7PdcryaAu0sAZbrN9i7WWcTMWvro=
-github.com/prometheus/common v0.4.0 h1:7etb9YClo3a6HjLzfl6rIQaU+FDfi0VSX39io3aQ+DM=
 github.com/prometheus/common v0.4.0/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4=
 github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
 github.com/prometheus/procfs v0.0.0-20190507164030-5867b95ac084/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA=
@ -258,15 +266,14 @@ github.com/rogpeppe/fastuuid v0.0.0-20150106093220-6724a57986af/go.mod h1:XWv6So
 github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
 github.com/russross/blackfriday/v2 v2.0.1/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
 github.com/ryanuber/columnize v0.0.0-20160712163229-9b3edd62028f/go.mod h1:sm1tb6uqfes/u+d4ooFouqFdy9/2g9QGwK3SQygK0Ts=
-github.com/sassoftware/go-rpmutils v0.1.1 h1:ZHMXpGoMHL/cpQJ1byyhEBW73TDtpNq4inYa9M4FxyY=
-github.com/sassoftware/go-rpmutils v0.1.1/go.mod h1:euhXULoBpvAxqrBHEyJS4Tsu3hHxUmQWNymxoJbzgUY=
 github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529/go.mod h1:DxrIzT+xaE7yg65j358z/aeFdxmN0P9QXhEzd20vsDc=
 github.com/sergi/go-diff v1.1.0 h1:we8PVUC3FE2uYfodKH/nBHMSetSfHDR6scGdBi+erh0=
 github.com/sergi/go-diff v1.1.0/go.mod h1:STckp+ISIX8hZLjrqAeVduY0gWCT9IjLuqbuNXdaHfM=
 github.com/shurcooL/sanitized_anchor_name v1.0.0/go.mod h1:1NzhyTcUVG4SuEtjjoZeVRXNmyL/1OwPU0+IJeTBvfc=
-github.com/sirupsen/logrus v1.2.0 h1:juTguoYk5qI21pwyTXY3B3Y5cOTH3ZUyZCg1v/mihuo=
 github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
+github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d h1:zE9ykElWQ6/NYmHa3jpm/yHnI4xSofP+UP6SpjHcSeM=
 github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc=
+github.com/smartystreets/goconvey v1.6.4 h1:fv0U8FUIMPNf1L9lnHLvLhgicrIVChEkdzIKYqbNC9s=
 github.com/smartystreets/goconvey v1.6.4/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA=
 github.com/soheilhy/cmux v0.1.4/go.mod h1:IM3LyeVVIOuxMH7sFAkER9+bJ4dT7Ms6E4xg4kGIyLM=
 github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA=
@ -287,14 +294,13 @@ github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+
 github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
 github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
 github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
+github.com/stretchr/testify v1.4.0 h1:2E4SXV/wtOkTonXsotYi4li6zVWxYlZuYNCXe9XRJyk=
 github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
 github.com/subosito/gotenv v1.2.0 h1:Slr1R9HxAlEKefgq5jn9U+DnETlIUa6HfgEzj0g5d7s=
 github.com/subosito/gotenv v1.2.0/go.mod h1:N0PQaV/YGNqwC0u51sEeR/aUtSLEXKX9iv69rRypqCw=
 github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
 github.com/xanzy/ssh-agent v0.2.1 h1:TCbipTQL2JiiCprBWx9frJ2eJlCYT00NmctrHxVAr70=
 github.com/xanzy/ssh-agent v0.2.1/go.mod h1:mLlQY/MoOhWBj+gOGMQkOeiEvkx+8pJSI+0Bx9h2kr4=
-github.com/xi2/xz v0.0.0-20171230120015-48954b6210f8 h1:nIPpBwaJSVYIxUFsDv3M8ofmx9yWTog9BfvIu0q41lo=
-github.com/xi2/xz v0.0.0-20171230120015-48954b6210f8/go.mod h1:HUYIGzjTL3rfEspMxjDjgmT5uz5wzYJKVo23qUhYTos=
 github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU=
 github.com/yuin/goldmark v1.1.25/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
 github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
@ -317,10 +323,7 @@ golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACk
 golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
 golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
 golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
-golang.org/x/crypto v0.0.0-20200302210943-78000ba7a073 h1:xMPOj6Pz6UipU1wXLkrtqpHbR0AVFnyPEQq/wRWz9lM=
 golang.org/x/crypto v0.0.0-20200302210943-78000ba7a073/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
-golang.org/x/crypto v0.0.0-20200604202706-70a84ac30bf9 h1:vEg9joUBmeBcK9iSJftGNf3coIG4HqZElCPehJsfAYM=
-golang.org/x/crypto v0.0.0-20200604202706-70a84ac30bf9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
 golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9 h1:psW17arqaxU48Z5kZ0CQnkZWQJsqcURM6tKiBApRjXI=
 golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
 golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
@ -376,7 +379,6 @@ golang.org/x/net v0.0.0-20200114155413-6afb5195e5aa/go.mod h1:z5CRVTTTmAJ677TzLL
 golang.org/x/net v0.0.0-20200202094626-16171245cfb2/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20200222125558-5a598a2470a0/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
-golang.org/x/net v0.0.0-20200301022130-244492dfa37a h1:GuSPYbZzB5/dcLNCwLQLsg3obCJtX9IJhpXkvY7kzk0=
 golang.org/x/net v0.0.0-20200301022130-244492dfa37a/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20200324143707-d3edc9973b7e/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
 golang.org/x/net v0.0.0-20200501053045-e0ff5e5a1de5/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
@ -386,7 +388,6 @@ golang.org/x/net v0.0.0-20200520182314-0ba52f642ac2/go.mod h1:qpuaurCH72eLCgpAm/
 golang.org/x/net v0.0.0-20200625001655-4c5254603344/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
 golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
 golang.org/x/net v0.0.0-20200822124328-c89045814202/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
-golang.org/x/net v0.0.0-20200904194848-62affa334b73 h1:MXfv8rhZWmFeqX3GNZRsd6vOLoaCHjYEX3qkRo3YBUA=
 golang.org/x/net v0.0.0-20200904194848-62affa334b73/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
 golang.org/x/net v0.0.0-20201110031124-69a78807bb2b h1:uwuIcX0g4Yl1NC5XAz37xsr2lTtcqevgzYNVt49waME=
 golang.org/x/net v0.0.0-20201110031124-69a78807bb2b/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
@ -428,7 +429,6 @@ golang.org/x/sys v0.0.0-20200122134326-e047566fdf82/go.mod h1:h1NjWce9XRLGQEsW7w
 golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200212091648-12a6c2dcc1e4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/sys v0.0.0-20200302150141-5c8b2ff67527 h1:uYVVQ9WP/Ds2ROhcaGPeIdVq0RIXVLwsHlnvJ+cT1So=
 golang.org/x/sys v0.0.0-20200302150141-5c8b2ff67527/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200331124033-c3d80250170d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
@ -438,10 +438,10 @@ golang.org/x/sys v0.0.0-20200515095857-1151b9dac4a9/go.mod h1:h1NjWce9XRLGQEsW7w
 golang.org/x/sys v0.0.0-20200523222454-059865788121/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200803210538-64077c9b5642/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200828194041-157a740278f4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/sys v0.0.0-20200905004654-be1d3432aa8f h1:Fqb3ao1hUmOR3GkUOg/Y+BadLwykBIzs5q8Ez2SbHyc=
 golang.org/x/sys v0.0.0-20200905004654-be1d3432aa8f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f h1:+Nyd8tzPX9R7BWHguqsrbFdRx3WQ/1ib8I44HXV5yTA=
 golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.1.0 h1:kunALQeHf1/185U1i0GOB/fy1IPRDDpuoOOqRReG57U=
+golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
@ -590,11 +590,11 @@ google.golang.org/protobuf v1.23.1-0.20200526195155-81db48ad09cc/go.mod h1:EGpAD
 google.golang.org/protobuf v1.24.0/go.mod h1:r/3tXBNzIEhYS9I1OUVjXDlt8tc493IdKGjtUeSXeh4=
 google.golang.org/protobuf v1.25.0 h1:Ejskq+SyPohKW+1uil0JJMtmHCgJPJ/qWTxr8qp+R4c=
 google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlbajtzgsN7c=
-gopkg.in/alecthomas/kingpin.v2 v2.2.6 h1:jMFz6MfLP0/4fUyZle81rXUoxOBFi19VUFKVDOQfozc=
 gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f h1:BLraFXnmrev5lT+xlilqcH8XK9/i0At2xKjWk4p6zsU=
 gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI=
 gopkg.in/ini.v1 v1.51.0 h1:AQvPpx3LzTDM0AjnIRlVFwFFGC+npRopjZxLJj6gdno=
@ -608,6 +608,8 @@ gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.2.8 h1:obN1ZagJSUGI0Ek/LBmuj4SNLPfIny3KsKFopxRdj10=
 gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
+gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b h1:h8qDotaEPuJATrMmW04NCwg7v22aHH28wwpauUhK9Oo=
+gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
 honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
 honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
 honnef.co/go/tools v0.0.0-20190418001031-e561f6794a2a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
--- a/modulemd/modulemd.go
+++ b/modulemd/modulemd.go
@ -21,122 +21,189 @@
 package modulemd

 import (
+	"fmt"
+
 	"github.com/go-git/go-billy/v5"
-	"gopkg.in/yaml.v2"
+	"gopkg.in/yaml.v3"
 )

+type ServiceLevelType string
+
+const (
+	ServiceLevelRawhide       ServiceLevelType = "rawhide"
+	ServiceLevelStableAPI     ServiceLevelType = "stable_api"
+	ServiceLevelBugFixes      ServiceLevelType = "bug_fixes"
+	ServiceLevelSecurityFixes ServiceLevelType = "security_fixes"
+)
+
+type ServiceLevel struct {
+	Eol string `yaml:"eol,omitempty"`
+}
+
+type License struct {
+	Module  []string `yaml:"module,omitempty"`
+	Content []string `yaml:"content,omitempty"`
+}
+
+type Dependencies struct {
+	BuildRequires map[string][]string `yaml:"buildrequires,omitempty,omitempty"`
+	Requires      map[string][]string `yaml:"requires,omitempty,omitempty"`
+}
+
+type References struct {
+	Community     string `yaml:"community,omitempty"`
+	Documentation string `yaml:"documentation,omitempty"`
+	Tracker       string `yaml:"tracker,omitempty"`
+}
+
+type Profile struct {
+	Description string   `yaml:"description,omitempty"`
+	Rpms        []string `yaml:"rpms,omitempty"`
+}
+
+type API struct {
+	Rpms []string `yaml:"rpms,omitempty"`
+}
+
+type BuildOptsRPM struct {
+	Macros    string   `yaml:"macros,omitempty"`
+	Whitelist []string `yaml:"whitelist,omitempty"`
+}
+
+type BuildOpts struct {
+	Rpms   *BuildOptsRPM `yaml:"rpms,omitempty"`
+	Arches []string      `yaml:"arches,omitempty"`
+}
+
+type ComponentRPM struct {
+	Name          string   `yaml:"name,omitempty"`
+	Rationale     string   `yaml:"rationale,omitempty"`
+	Repository    string   `yaml:"repository,omitempty"`
+	Cache         string   `yaml:"cache,omitempty"`
+	Ref           string   `yaml:"ref,omitempty"`
+	Buildonly     bool     `yaml:"buildonly,omitempty"`
+	Buildroot     bool     `yaml:"buildroot,omitempty"`
+	SrpmBuildroot bool     `yaml:"srpm-buildroot,omitempty"`
+	Buildorder    int      `yaml:"buildorder,omitempty"`
+	Arches        []string `yaml:"arches,omitempty"`
+	Multilib      []string `yaml:"multilib,omitempty"`
+}
+
+type ComponentModule struct {
+	Rationale  string `yaml:"rationale,omitempty"`
+	Repository string `yaml:"repository,omitempty"`
+	Ref        string `yaml:"ref,omitempty"`
+	Buildorder int    `yaml:"buildorder,omitempty"`
+}
+
+type Components struct {
+	Rpms    map[string]*ComponentRPM    `yaml:"rpms,omitempty"`
+	Modules map[string]*ComponentModule `yaml:"modules,omitempty"`
+}
+
+type ArtifactsRPMMap struct {
+	Name    string  `yaml:"name,omitempty"`
+	Epoch   int     `yaml:"epoch,omitempty"`
+	Version float64 `yaml:"version,omitempty"`
+	Release string  `yaml:"release,omitempty"`
+	Arch    string  `yaml:"arch,omitempty"`
+	Nevra   string  `yaml:"nevra,omitempty"`
+}
+
+type Artifacts struct {
+	Rpms   []string                               `yaml:"rpms,omitempty"`
+	RpmMap map[string]map[string]*ArtifactsRPMMap `yaml:"rpm-map,omitempty"`
+}
+
+type Data struct {
+	Name          string                             `yaml:"name,omitempty"`
+	Stream        string                             `yaml:"stream,omitempty"`
+	Version       string                             `yaml:"version,omitempty"`
+	StaticContext bool                               `yaml:"static_context,omitempty"`
+	Context       string                             `yaml:"context,omitempty"`
+	Arch          string                             `yaml:"arch,omitempty"`
+	Summary       string                             `yaml:"summary,omitempty"`
+	Description   string                             `yaml:"description,omitempty"`
+	ServiceLevels map[ServiceLevelType]*ServiceLevel `yaml:"servicelevels,omitempty"`
+	License       *License                           `yaml:"license,omitempty"`
+	Xmd           map[string]map[string]string       `yaml:"xmd,omitempty"`
+	Dependencies  []*Dependencies                    `yaml:"dependencies,omitempty"`
+	References    *References                        `yaml:"references,omitempty"`
+	Profiles      map[string]*Profile                `yaml:"profiles,omitempty"`
+	Profile       map[string]*Profile                `yaml:"profile,omitempty"`
+	API           *API                               `yaml:"api,omitempty"`
+	Filter        *API                               `yaml:"filter,omitempty"`
+	BuildOpts     *BuildOpts                         `yaml:"buildopts,omitempty"`
+	Components    *Components                        `yaml:"components,omitempty"`
+	Artifacts     *Artifacts                         `yaml:"artifacts,omitempty"`
+}
+
 type ModuleMd struct {
 	Document string `yaml:"document,omitempty"`
 	Version  int    `yaml:"version,omitempty"`
-	Data     struct {
-		Name          string `yaml:"name,omitempty"`
-		Stream        string `yaml:"stream,omitempty"`
-		Version       int64  `yaml:"version,omitempty"`
-		StaticContext bool   `yaml:"static_context,omitempty"`
-		Context       string `yaml:"context,omitempty"`
-		Arch          string `yaml:"arch,omitempty"`
-		Summary       string `yaml:"summary,omitempty"`
-		Description   string `yaml:"description,omitempty"`
-		Servicelevels struct {
-			Rawhide struct {
-				Eol struct {
-				} `yaml:"eol,omitempty"`
-			} `yaml:"rawhide,omitempty"`
-			StableAPI struct {
-				Eol struct {
-				} `yaml:"eol,omitempty"`
-			} `yaml:"stable_api,omitempty"`
-			BugFixes struct {
-				Eol struct {
-				} `yaml:"eol,omitempty"`
-			} `yaml:"bug_fixes,omitempty"`
-			SecurityFixes struct {
-				Eol struct {
-				} `yaml:"eol,omitempty"`
-			} `yaml:"security_fixes,omitempty"`
-		} `yaml:"servicelevels,omitempty"`
-		License struct {
-			Module  []string `yaml:"module,omitempty"`
-			Content []string `yaml:"content,omitempty"`
-		} `yaml:"license,omitempty"`
-		Xmd          map[string]interface{} `yaml:"xmd,omitempty"`
-		Dependencies []struct {
-			Buildrequires map[string][]string `yaml:"buildrequires,omitempty,omitempty"`
-			Requires      map[string][]string `yaml:"requires,omitempty,omitempty"`
-		} `yaml:"dependencies,omitempty"`
-		References struct {
-			Community     string `yaml:"community,omitempty"`
-			Documentation string `yaml:"documentation,omitempty"`
-			Tracker       string `yaml:"tracker,omitempty"`
-		} `yaml:"references,omitempty"`
-		Profiles map[string]*struct {
-			Description string   `yaml:"description,omitempty"`
-			Rpms        []string `yaml:"rpms,omitempty"`
-		} `yaml:"profiles,omitempty"`
-		Profile map[string]*struct {
-			Description string   `yaml:"description,omitempty"`
-			Rpms        []string `yaml:"rpms,omitempty"`
-		} `yaml:"profile,omitempty"`
-		API struct {
-			Rpms []string `yaml:"rpms,omitempty"`
-		} `yaml:"api,omitempty"`
-		Filter struct {
-			Rpms []string `yaml:"rpms,omitempty"`
-		} `yaml:"filter,omitempty"`
-		Buildopts struct {
-			Rpms struct {
-				Macros    string   `yaml:"macros,omitempty"`
-				Whitelist []string `yaml:"whitelist,omitempty"`
-			} `yaml:"rpms,omitempty"`
-			Arches []string `yaml:"arches,omitempty"`
-		} `yaml:"buildopts,omitempty"`
-		Components struct {
-			Rpms map[string]*struct {
-				Name          string   `yaml:"name,omitempty"`
-				Rationale     string   `yaml:"rationale,omitempty"`
-				Repository    string   `yaml:"repository,omitempty"`
-				Cache         string   `yaml:"cache,omitempty"`
-				Ref           string   `yaml:"ref,omitempty"`
-				Buildonly     bool     `yaml:"buildonly,omitempty"`
-				Buildroot     bool     `yaml:"buildroot,omitempty"`
-				SrpmBuildroot bool     `yaml:"srpm-buildroot,omitempty"`
-				Buildorder    int      `yaml:"buildorder,omitempty"`
-				Arches        []string `yaml:"arches,omitempty"`
-				Multilib      []string `yaml:"multilib,omitempty"`
-			} `yaml:"rpms,omitempty"`
-			Modules map[string]*struct {
-				Rationale  string `yaml:"rationale,omitempty"`
-				Repository string `yaml:"repository,omitempty"`
-				Ref        string `yaml:"ref,omitempty"`
-				Buildorder int    `yaml:"buildorder,omitempty"`
-			} `yaml:"modules,omitempty"`
-		} `yaml:"components,omitempty"`
-		Artifacts struct {
-			Rpms   []string `yaml:"rpms,omitempty"`
-			RpmMap map[string]map[string]*struct {
-				Name    string  `yaml:"name,omitempty"`
-				Epoch   int     `yaml:"epoch,omitempty"`
-				Version float64 `yaml:"version,omitempty"`
-				Release string  `yaml:"release,omitempty"`
-				Arch    string  `yaml:"arch,omitempty"`
-				Nevra   string  `yaml:"nevra,omitempty"`
-			} `yaml:"rpm-map,omitempty"`
-		} `yaml:"artifacts,omitempty"`
-	} `yaml:"data,omitempty"`
+	Data     *Data  `yaml:"data,omitempty"`
 }

-func Parse(input []byte) (*ModuleMd, error) {
-	var ret ModuleMd
-	err := yaml.Unmarshal(input, &ret)
+type DetectVersionDocument struct {
+	Document string `yaml:"document,omitempty"`
+	Version  int    `yaml:"version,omitempty"`
+}
+
+type DefaultsData struct {
+	Module   string              `yaml:"module,omitempty"`
+	Stream   string              `yaml:"stream,omitempty"`
+	Profiles map[string][]string `yaml:"profiles,omitempty"`
+}
+
+type Defaults struct {
+	Document string        `yaml:"document,omitempty"`
+	Version  int           `yaml:"version,omitempty"`
+	Data     *DefaultsData `yaml:"data,omitempty"`
+}
+
+type NotBackwardsCompatibleModuleMd struct {
+	V2 *ModuleMd
+	V3 *V3
+}
+
+func Parse(input []byte) (*NotBackwardsCompatibleModuleMd, error) {
+	var detect DetectVersionDocument
+	err := yaml.Unmarshal(input, &detect)
 	if err != nil {
-		return nil, err
+		return nil, fmt.Errorf("error detecting document version: %s", err)
+	}
+
+	var ret NotBackwardsCompatibleModuleMd
+
+	if detect.Version == 2 {
+		var v2 ModuleMd
+		err = yaml.Unmarshal(input, &v2)
+		if err != nil {
+			return nil, fmt.Errorf("error parsing modulemd: %s", err)
+		}
+		ret.V2 = &v2
+	} else if detect.Version == 3 {
+		var v3 V3
+		err = yaml.Unmarshal(input, &v3)
+		if err != nil {
+			return nil, fmt.Errorf("error parsing modulemd: %s", err)
+		}
+		ret.V3 = &v3
 	}

 	return &ret, nil
 }

-func (m *ModuleMd) Marshal(fs billy.Filesystem, path string) error {
-	bts, err := yaml.Marshal(m)
+func (m *NotBackwardsCompatibleModuleMd) Marshal(fs billy.Filesystem, path string) error {
+	var bts []byte
+
+	var err error
+	if m.V2 != nil {
+		bts, err = yaml.Marshal(m.V2)
+	}
+	if m.V3 != nil {
+		bts, err = yaml.Marshal(m.V3)
+	}
 	if err != nil {
 		return err
 	}
--- a/modulemd/v3.go
+++ b/modulemd/v3.go
@ -0,0 +1,32 @@
+package modulemd
+
+type V3 struct {
+	Document string  `yaml:"document,omitempty"`
+	Version  int     `yaml:"version,omitempty"`
+	Data     *V3Data `yaml:"data,omitempty"`
+}
+
+type Configurations struct {
+	Context       string              `yaml:"context,omitempty"`
+	Platform      string              `yaml:"platform,omitempty"`
+	BuildRequires map[string][]string `yaml:"buildrequires,omitempty"`
+	Requires      map[string][]string `yaml:"requires,omitempty"`
+	BuildOpts     *BuildOpts          `yaml:"buildopts,omitempty"`
+}
+
+type V3Data struct {
+	Name           string                       `yaml:"name,omitempty"`
+	Stream         string                       `yaml:"stream,omitempty"`
+	Summary        string                       `yaml:"summary,omitempty"`
+	Description    string                       `yaml:"description,omitempty"`
+	License        []string                     `yaml:"license,omitempty"`
+	Xmd            map[string]map[string]string `yaml:"xmd,omitempty"`
+	Configurations []*Configurations            `yaml:"configurations,omitempty"`
+	References     *References                  `yaml:"references,omitempty"`
+	Profiles       map[string]*Profile          `yaml:"profiles,omitempty"`
+	Profile        map[string]*Profile          `yaml:"profile,omitempty"`
+	API            *API                         `yaml:"api,omitempty"`
+	Filter         *API                         `yaml:"filter,omitempty"`
+	Demodularized  *API                         `yaml:"demodularized,omitempty"`
+	Components     *Components                  `yaml:"components,omitempty"`
+}
--- a/pkg/blob/file/file.go
+++ b/pkg/blob/file/file.go
@ -38,7 +38,7 @@ func New(path string) *File {
 }

 func (f *File) Write(path string, content []byte) error {
-	w, err := os.OpenFile(filepath.Join(f.path, path), os.O_CREATE|os.O_TRUNC|os.O_RDWR, 0644)
+	w, err := os.OpenFile(filepath.Join(f.path, path), os.O_CREATE|os.O_TRUNC|os.O_RDWR, 0o644)
 	if err != nil {
 		return fmt.Errorf("could not open file: %v", err)
 	}
@ -57,7 +57,7 @@ func (f *File) Write(path string, content []byte) error {
 }

 func (f *File) Read(path string) ([]byte, error) {
-	r, err := os.OpenFile(filepath.Join(f.path, path), os.O_RDONLY, 0644)
+	r, err := os.OpenFile(filepath.Join(f.path, path), os.O_RDONLY, 0o644)
 	if err != nil {
 		if os.IsNotExist(err) {
 			return nil, nil
--- a/pkg/blob/gcs/gcs.go
+++ b/pkg/blob/gcs/gcs.go
@ -21,10 +21,11 @@
 package gcs

 import (
-	"cloud.google.com/go/storage"
 	"context"
 	"fmt"
 	"io/ioutil"
+
+	"cloud.google.com/go/storage"
 )

 type GCS struct {
--- a/pkg/blob/s3/s3.go
+++ b/pkg/blob/s3/s3.go
@ -22,6 +22,8 @@ package s3

 import (
 	"bytes"
+	"io/ioutil"
+
 	"github.com/aws/aws-sdk-go/aws"
 	"github.com/aws/aws-sdk-go/aws/awserr"
 	"github.com/aws/aws-sdk-go/aws/credentials"
@ -29,7 +31,6 @@ import (
 	"github.com/aws/aws-sdk-go/service/s3"
 	"github.com/aws/aws-sdk-go/service/s3/s3manager"
 	"github.com/spf13/viper"
-	"io/ioutil"
 )

 type S3 struct {
--- a/pkg/data/import.go
+++ b/pkg/data/import.go
@ -21,8 +21,9 @@
 package data

 import (
-	"github.com/go-git/go-git/v5"
 	"hash"
+
+	"github.com/go-git/go-git/v5"
 )

 type ImportMode interface {
--- a/pkg/data/process.go
+++ b/pkg/data/process.go
@ -21,10 +21,11 @@
 package data

 import (
+	"log"
+
 	"github.com/go-git/go-billy/v5"
 	"github.com/go-git/go-git/v5/plumbing/transport"
 	"github.com/rocky-linux/srpmproc/pkg/blob"
-	"log"
 )

 type FsCreatorFunc func(branch string) (billy.Filesystem, error)
@ -50,8 +51,14 @@ type ProcessData struct {
 	NoStorageUpload      bool
 	ManualCommits        []string
 	ModuleFallbackStream string
-	AllowStreamBranches  bool
+	BranchSuffix         string
+	StrictBranchMode     bool
 	FsCreator            FsCreatorFunc
 	CdnUrl               string
 	Log                  *log.Logger
+	PackageVersion       string
+	PackageRelease       string
+	TaglessMode          bool
+	Cdn                  string
+	ModuleBranchNames    bool
 }
--- a/pkg/data/utils.go
+++ b/pkg/data/utils.go
@ -27,11 +27,12 @@ import (
 	"crypto/sha512"
 	"encoding/hex"
 	"fmt"
-	"github.com/go-git/go-billy/v5"
 	"hash"
 	"io"
 	"os"
 	"path/filepath"
+
+	"github.com/go-git/go-billy/v5"
 )

 func CopyFromFs(from billy.Filesystem, to billy.Filesystem, path string) error {
@ -44,7 +45,7 @@ func CopyFromFs(from billy.Filesystem, to billy.Filesystem, path string) error {
 		fullPath := filepath.Join(path, fi.Name())

 		if fi.IsDir() {
-			_ = to.MkdirAll(fullPath, 0755)
+			_ = to.MkdirAll(fullPath, 0o755)
 			err := CopyFromFs(from, to, fullPath)
 			if err != nil {
 				return err
--- a/pkg/directives/add.go
+++ b/pkg/directives/add.go
@ -50,7 +50,7 @@ func add(cfg *srpmprocpb.Cfg, pd *data.ProcessData, md *data.ModeData, patchTree
 		case *srpmprocpb.Add_File:
 			filePath = checkAddPrefix(eitherString(filepath.Base(addType.File), add.Name))

-			fPatch, err := patchTree.Filesystem.OpenFile(addType.File, os.O_RDONLY, 0644)
+			fPatch, err := patchTree.Filesystem.OpenFile(addType.File, os.O_RDONLY, 0o644)
 			if err != nil {
 				return errors.New(fmt.Sprintf("COULD_NOT_OPEN_FROM:%s", addType.File))
 			}
@ -80,7 +80,7 @@ func add(cfg *srpmprocpb.Cfg, pd *data.ProcessData, md *data.ModeData, patchTree
 			break
 		}

-		f, err := pushTree.Filesystem.OpenFile(filePath, os.O_RDWR|os.O_CREATE|os.O_TRUNC, 0644)
+		f, err := pushTree.Filesystem.OpenFile(filePath, os.O_RDWR|os.O_CREATE|os.O_TRUNC, 0o644)
 		if err != nil {
 			return errors.New(fmt.Sprintf("COULD_NOT_OPEN_DESTINATION:%s", filePath))
 		}
--- a/pkg/directives/lookaside.go
+++ b/pkg/directives/lookaside.go
@ -128,7 +128,7 @@ func lookaside(cfg *srpmprocpb.Cfg, _ *data.ProcessData, md *data.ModeData, patc
 			}

 			path := filepath.Join("SOURCES", fmt.Sprintf("%s.tar.gz", directive.ArchiveName))
-			pushF, err := pushTree.Filesystem.OpenFile(path, os.O_CREATE|os.O_TRUNC|os.O_RDWR, 0644)
+			pushF, err := pushTree.Filesystem.OpenFile(path, os.O_CREATE|os.O_TRUNC|os.O_RDWR, 0o644)
 			if err != nil {
 				return errors.New(fmt.Sprintf("COULD_NOT_CREATE_TAR_FILE:%s", path))
 			}
--- a/pkg/directives/patch.go
+++ b/pkg/directives/patch.go
@ -24,6 +24,7 @@ import (
 	"bytes"
 	"errors"
 	"fmt"
+
 	"github.com/bluekeyes/go-gitdiff/gitdiff"
 	"github.com/go-git/go-git/v5"
 	srpmprocpb "github.com/rocky-linux/srpmproc/pb"
--- a/pkg/directives/replace.go
+++ b/pkg/directives/replace.go
@ -51,7 +51,7 @@ func replace(cfg *srpmprocpb.Cfg, pd *data.ProcessData, _ *data.ModeData, patchT

 		switch replacing := replace.Replacing.(type) {
 		case *srpmprocpb.Replace_WithFile:
-			fPatch, err := patchTree.Filesystem.OpenFile(replacing.WithFile, os.O_RDONLY, 0644)
+			fPatch, err := patchTree.Filesystem.OpenFile(replacing.WithFile, os.O_RDONLY, 0o644)
 			if err != nil {
 				return errors.New(fmt.Sprintf("COULD_NOT_OPEN_REPLACING:%s", replacing.WithFile))
 			}
--- a/pkg/directives/spec_change.go
+++ b/pkg/directives/spec_change.go
@ -41,9 +41,7 @@ const (
 	sectionChangelog = "%changelog"
 )

-var (
-	sections = []string{"%description", "%prep", "%build", "%install", "%files", "%changelog"}
-)
+var sections = []string{"%description", "%prep", "%build", "%install", "%files", "%changelog"}

 type sourcePatchOperationInLoopRequest struct {
 	cfg           *srpmprocpb.Cfg
@ -212,7 +210,7 @@ func specChange(cfg *srpmprocpb.Cfg, pd *data.ProcessData, md *data.ModeData, _
 		return errors.New("COULD_NOT_STAT_SPEC_FILE")
 	}

-	specFile, err := pushTree.Filesystem.OpenFile(filePath, os.O_RDONLY, 0644)
+	specFile, err := pushTree.Filesystem.OpenFile(filePath, os.O_RDONLY, 0o644)
 	if err != nil {
 		return errors.New("COULD_NOT_READ_SPEC_FILE")
 	}
--- a/pkg/misc/regex.go
+++ b/pkg/misc/regex.go
@ -2,13 +2,52 @@ package misc

 import (
 	"fmt"
+	"path/filepath"
 	"regexp"
+	"strings"
+
+	"github.com/rocky-linux/srpmproc/pkg/data"
 )

-func GetTagImportRegex(importBranchPrefix string, allowStreamBranches bool) *regexp.Regexp {
-	if allowStreamBranches {
-		return regexp.MustCompile(fmt.Sprintf("refs/tags/(imports/(%s(?:.s|.)|%s(?:|s).+)/(.*))", importBranchPrefix, importBranchPrefix))
+func GetTagImportRegex(pd *data.ProcessData) *regexp.Regexp {
+	branchRegex := regexp.QuoteMeta(fmt.Sprintf("%s%d%s", pd.ImportBranchPrefix, pd.Version, pd.BranchSuffix))
+	if !pd.StrictBranchMode {
+		branchRegex += "(?:.+|)"
 	} else {
-		return regexp.MustCompile(fmt.Sprintf("refs/tags/(imports/(%s.|%s.-.+)/(.*))", importBranchPrefix, importBranchPrefix))
+		branchRegex += "(?:-stream-.+|)"
 	}
+
+	initialVerRegex := regexp.QuoteMeta(filepath.Base(pd.RpmLocation)) + "-"
+	if pd.PackageVersion != "" {
+		initialVerRegex += regexp.QuoteMeta(pd.PackageVersion) + "-"
+	} else {
+		initialVerRegex += ".+-"
+	}
+	if pd.PackageRelease != "" {
+		initialVerRegex += regexp.QuoteMeta(pd.PackageRelease)
+	} else {
+		initialVerRegex += ".+"
+	}
+
+	regex := fmt.Sprintf("(?i)refs/tags/(imports/(%s)/(%s))", branchRegex, initialVerRegex)
+
+	return regexp.MustCompile(regex)
+}
+
+// Given a git reference in tagless mode (like "refs/heads/c9s", or "refs/heads/stream-httpd-2.4-rhel-9.1.0"), determine
+// if we are ok with importing that reference.  We are looking for the traditional <prefix><version><suffix> pattern, like "c9s", and also the
+// modular "stream-<NAME>-<VERSION>-rhel-<VERSION> branch pattern as well
+func TaglessRefOk(tag string, pd *data.ProcessData) bool {
+	// First case is very easy: if we are exactly "refs/heads/<prefix><version><suffix>" , then this is def. a branch we should import
+	if tag == fmt.Sprintf("refs/heads/%s%d%s", pd.ImportBranchPrefix, pd.Version, pd.BranchSuffix) {
+		return true
+	}
+
+	// Less easy: if a modular branch is present (starts w/ "stream-"), we need to check if it's part of our major version, and return true if it is
+	// (major version means we look for the text "rhel-X." in the branch name, like "rhel-9.1.0")
+	if strings.HasPrefix(tag, "refs/heads/stream-") && strings.Contains(tag, fmt.Sprintf("rhel-%d.", pd.Version)) {
+		return true
+	}
+
+	return false
 }
--- a/pkg/modes/git.go
+++ b/pkg/modes/git.go
@ -21,16 +21,20 @@
 package modes

 import (
+	"bytes"
 	"fmt"
-	"github.com/go-git/go-git/v5/plumbing/transport"
-	"github.com/rocky-linux/srpmproc/pkg/misc"
 	"io/ioutil"
+	"log"
 	"net/http"
 	"path/filepath"
 	"sort"
 	"strings"
+	"text/template"
 	"time"

+	"github.com/go-git/go-git/v5/plumbing/transport"
+	"github.com/rocky-linux/srpmproc/pkg/misc"
+
 	"github.com/go-git/go-billy/v5/memfs"
 	"github.com/go-git/go-git/v5"
 	"github.com/go-git/go-git/v5/config"
@ -45,6 +49,15 @@ type remoteTarget struct {
 	when   time.Time
 }

+// Struct to define the possible template values ( {{.Value}} in CDN URL strings:
+type Lookaside struct {
+	Name     string
+	Branch   string
+	Hash     string
+	Hashtype string
+	Filename string
+}
+
 type remoteTargetSlice []remoteTarget

 func (p remoteTargetSlice) Len() int {
@ -109,14 +122,13 @@ func (g *GitMode) RetrieveSource(pd *data.ProcessData) (*data.ModeData, error) {
 	tagAdd := func(tag *object.Tag) error {
 		if strings.HasPrefix(tag.Name, fmt.Sprintf("imports/%s%d", pd.ImportBranchPrefix, pd.Version)) {
 			refSpec := fmt.Sprintf("refs/tags/%s", tag.Name)
-			if misc.GetTagImportRegex(pd.ImportBranchPrefix, pd.AllowStreamBranches).MatchString(refSpec) {
-				match := misc.GetTagImportRegex(pd.ImportBranchPrefix, pd.AllowStreamBranches).FindStringSubmatch(refSpec)
+			if misc.GetTagImportRegex(pd).MatchString(refSpec) {
+				match := misc.GetTagImportRegex(pd).FindStringSubmatch(refSpec)

 				exists := latestTags[match[2]]
 				if exists != nil && exists.when.After(tag.Tagger.When) {
 					return nil
 				}
-
 				latestTags[match[2]] = &remoteTarget{
 					remote: refSpec,
 					when:   tag.Tagger.When,
@ -126,11 +138,38 @@ func (g *GitMode) RetrieveSource(pd *data.ProcessData) (*data.ModeData, error) {
 		return nil
 	}

+	// In case of "tagless mode", we need to get the head ref of the branch instead
+	// This is a kind of alternative implementation of the above tagAdd assignment
+	refAdd := func(tag *object.Tag) error {
+		if misc.TaglessRefOk(tag.Name, pd) {
+			pd.Log.Printf("Tagless mode:  Identified tagless commit for import: %s\n", tag.Name)
+			refSpec := fmt.Sprintf(tag.Name)
+
+			// We split the string by "/", the branch name we're looking for to pass to latestTags is always last
+			// (ex: "refs/heads/c9s" ---> we want latestTags[c9s]
+			tmpRef := strings.Split(refSpec, "/")
+			tmpBranchName := tmpRef[(len(tmpRef) - 1)]
+
+			latestTags[tmpBranchName] = &remoteTarget{
+				remote: refSpec,
+				when:   tag.Tagger.When,
+			}
+		}
+		return nil
+	}
+
 	tagIter, err := repo.TagObjects()
 	if err != nil {
 		return nil, fmt.Errorf("could not get tag objects: %v", err)
 	}
-	_ = tagIter.ForEach(tagAdd)
+
+	// tagless mode means we use "refAdd" (add commit by reference)
+	// normal mode means we can rely on "tagAdd" (the tag should be present for us in the source repo)
+	if pd.TaglessMode {
+		_ = tagIter.ForEach(refAdd)
+	} else {
+		_ = tagIter.ForEach(tagAdd)
+	}

 	listOpts := &git.ListOptions{
 		Auth: pd.Authenticator,
@ -157,17 +196,26 @@ func (g *GitMode) RetrieveSource(pd *data.ProcessData) (*data.ModeData, error) {
 		if err != nil {
 			continue
 		}
-		_ = tagAdd(&object.Tag{
-			Name:   strings.TrimPrefix(string(ref.Name()), "refs/tags/"),
-			Tagger: commit.Committer,
-		})
+
+		// Call refAdd instead of tagAdd in the case of TaglessMode enabled
+		if pd.TaglessMode {
+			_ = refAdd(&object.Tag{
+				Name:   string(ref.Name()),
+				Tagger: commit.Committer,
+			})
+		} else {
+			_ = tagAdd(&object.Tag{
+				Name:   strings.TrimPrefix(string(ref.Name()), "refs/tags/"),
+				Tagger: commit.Committer,
+			})
+		}
+
 	}

 	for _, branch := range latestTags {
 		pd.Log.Printf("tag: %s", strings.TrimPrefix(branch.remote, "refs/tags/"))
 		branches = append(branches, *branch)
 	}
-
 	sort.Sort(branches)

 	var sortedBranches []string
@ -186,56 +234,84 @@ func (g *GitMode) RetrieveSource(pd *data.ProcessData) (*data.ModeData, error) {

 func (g *GitMode) WriteSource(pd *data.ProcessData, md *data.ModeData) error {
 	remote, err := md.Repo.Remote("upstream")
-	if err != nil {
+
+	if err != nil && !pd.TaglessMode {
 		return fmt.Errorf("could not get upstream remote: %v", err)
 	}

 	var refspec config.RefSpec
 	var branchName string

-	if strings.HasPrefix(md.TagBranch, "refs/heads") {
-		refspec = config.RefSpec(fmt.Sprintf("+%s:%s", md.TagBranch, md.TagBranch))
-		branchName = strings.TrimPrefix(md.TagBranch, "refs/heads/")
-	} else {
-		match := misc.GetTagImportRegex(pd.ImportBranchPrefix, pd.AllowStreamBranches).FindStringSubmatch(md.TagBranch)
-		branchName = match[2]
-		refspec = config.RefSpec(fmt.Sprintf("+refs/heads/%s:%s", branchName, md.TagBranch))
-	}
-	pd.Log.Printf("checking out upstream refspec %s", refspec)
-	fetchOpts := &git.FetchOptions{
-		Auth:       pd.Authenticator,
-		RemoteName: "upstream",
-		RefSpecs:   []config.RefSpec{refspec},
-		Tags:       git.AllTags,
-		Force:      true,
-	}
-	err = remote.Fetch(fetchOpts)
-	if err != nil && err != git.NoErrAlreadyUpToDate {
-		if err == transport.ErrInvalidAuthMethod || err == transport.ErrAuthenticationRequired {
-			fetchOpts.Auth = nil
-			err = remote.Fetch(fetchOpts)
-			if err != nil && err != git.NoErrAlreadyUpToDate {
+	// In the case of tagless mode, we already have the transformed repo sitting in the worktree,
+	// and don't need to perform any checkout or fetch operations
+	if !pd.TaglessMode {
+		if strings.HasPrefix(md.TagBranch, "refs/heads") {
+			refspec = config.RefSpec(fmt.Sprintf("+%s:%s", md.TagBranch, md.TagBranch))
+			branchName = strings.TrimPrefix(md.TagBranch, "refs/heads/")
+		} else {
+			match := misc.GetTagImportRegex(pd).FindStringSubmatch(md.TagBranch)
+			branchName = match[2]
+			refspec = config.RefSpec(fmt.Sprintf("+refs/heads/%s:%s", branchName, md.TagBranch))
+			fmt.Println("Found branchname that does not start w/ refs/heads :: ", branchName)
+		}
+		pd.Log.Printf("checking out upstream refspec %s", refspec)
+
+		fetchOpts := &git.FetchOptions{
+			Auth:       pd.Authenticator,
+			RemoteName: "upstream",
+			RefSpecs:   []config.RefSpec{refspec},
+			Tags:       git.AllTags,
+			Force:      true,
+		}
+		err = remote.Fetch(fetchOpts)
+		if err != nil && err != git.NoErrAlreadyUpToDate {
+			if err == transport.ErrInvalidAuthMethod || err == transport.ErrAuthenticationRequired {
+				fetchOpts.Auth = nil
+				err = remote.Fetch(fetchOpts)
+				if err != nil && err != git.NoErrAlreadyUpToDate {
+					return fmt.Errorf("could not fetch upstream: %v", err)
+				}
+			} else {
 				return fmt.Errorf("could not fetch upstream: %v", err)
 			}
-		} else {
-			return fmt.Errorf("could not fetch upstream: %v", err)
+		}
+
+		err = md.Worktree.Checkout(&git.CheckoutOptions{
+			Branch: plumbing.ReferenceName(md.TagBranch),
+			Force:  true,
+		})
+		if err != nil {
+			return fmt.Errorf("could not checkout source from git: %v", err)
+		}
+
+		_, err = md.Worktree.Add(".")
+		if err != nil {
+			return fmt.Errorf("could not add Worktree: %v", err)
 		}
 	}

-	err = md.Worktree.Checkout(&git.CheckoutOptions{
-		Branch: plumbing.ReferenceName(md.TagBranch),
-		Force:  true,
-	})
-	if err != nil {
-		return fmt.Errorf("could not checkout source from git: %v", err)
+	if pd.TaglessMode {
+		branchName = fmt.Sprintf("%s%d%s", pd.ImportBranchPrefix, pd.Version, pd.BranchSuffix)
 	}

-	_, err = md.Worktree.Add(".")
+	metadataPath := ""
+	ls, err := md.Worktree.Filesystem.ReadDir(".")
 	if err != nil {
-		return fmt.Errorf("could not add Worktree: %v", err)
+		return fmt.Errorf("could not read directory: %v", err)
+	}
+	for _, f := range ls {
+		if strings.HasSuffix(f.Name(), ".metadata") {
+			if metadataPath != "" {
+				return fmt.Errorf("multiple metadata files found")
+			}
+			metadataPath = f.Name()
+		}
+	}
+	if metadataPath == "" {
+		metadataPath = fmt.Sprintf(".%s.metadata", md.Name)
 	}

-	metadataFile, err := md.Worktree.Filesystem.Open(fmt.Sprintf(".%s.metadata", md.Name))
+	metadataFile, err := md.Worktree.Filesystem.Open(metadataPath)
 	if err != nil {
 		pd.Log.Printf("warn: could not open metadata file, so skipping: %v", err)
 		return nil
@ -275,21 +351,68 @@ func (g *GitMode) WriteSource(pd *data.ProcessData, md *data.ModeData) error {
 				body = fromBlobStorage
 				pd.Log.Printf("downloading %s from blob storage", hash)
 			} else {
-				url := fmt.Sprintf("%s/%s/%s/%s", pd.CdnUrl, md.Name, branchName, hash)
-				pd.Log.Printf("downloading %s", url)

-				req, err := http.NewRequest("GET", url, nil)
-				if err != nil {
-					return fmt.Errorf("could not create new http request: %v", err)
-				}
-				req.Header.Set("Accept-Encoding", "*")
+				url := ""

-				resp, err := client.Do(req)
-				if err != nil {
-					return fmt.Errorf("could not download dist-git file: %v", err)
+				// We need to figure out the hashtype for templating purposes:
+				hashType := "sha512"
+				switch len(hash) {
+				case 128:
+					hashType = "sha512"
+				case 64:
+					hashType = "sha256"
+				case 40:
+					hashType = "sha1"
+				case 32:
+					hashType = "md5"
 				}
-				if resp.StatusCode != http.StatusOK {
+
+				// need the name of the file without "SOURCES/":
+				fileName := strings.Split(path, "/")[1]
+
+				// Feed our template info to ProcessUrl and transform to the real values: ( {{.Name}}, {{.Branch}}, {{.Hash}}, {{.Hashtype}}, {{.Filename}} )
+				url, hasTemplate := ProcessUrl(pd.CdnUrl, md.Name, branchName, hash, hashType, fileName)
+
+				var req *http.Request
+				var resp *http.Response
+
+				// Download the --cdn-url given, but *only* if it contains template strings ( {{.Name}} , {{.Hash}} , etc. )
+				// Otherwise we need to fall back to the traditional cdn-url patterns
+				if hasTemplate {
+					pd.Log.Printf("downloading %s", url)
+
+					req, err := http.NewRequest("GET", url, nil)
+					if err != nil {
+						return fmt.Errorf("could not create new http request: %v", err)
+					}
+					req.Header.Set("Accept-Encoding", "*")
+
+					resp, err = client.Do(req)
+					if err != nil {
+						return fmt.Errorf("could not download dist-git file: %v", err)
+					}
+				}
+
+				// Default cdn-url:  If we don't have a templated download string, try the default <SITE>/<PKG>/<BRANCH>/<HASH> pattern:
+				if resp == nil || resp.StatusCode != http.StatusOK {
+					url = fmt.Sprintf("%s/%s/%s/%s", pd.CdnUrl, md.Name, branchName, hash)
+					pd.Log.Printf("Attempting default URL: %s", url)
+					req, err = http.NewRequest("GET", url, nil)
+					if err != nil {
+						return fmt.Errorf("could not create new http request: %v", err)
+					}
+					req.Header.Set("Accept-Encoding", "*")
+					resp, err = client.Do(req)
+					if err != nil {
+						return fmt.Errorf("could not download dist-git file: %v", err)
+					}
+				}
+
+				// If the default URL fails, we have one more pattern to try.  The simple <SITE>/<HASH> pattern
+				// If this one fails, we are truly lost, and have to bail out w/ an error:
+				if resp == nil || resp.StatusCode != http.StatusOK {
 					url = fmt.Sprintf("%s/%s", pd.CdnUrl, hash)
+					pd.Log.Printf("Attempting 2nd fallback URL: %s", url)
 					req, err = http.NewRequest("GET", url, nil)
 					if err != nil {
 						return fmt.Errorf("could not create new http request: %v", err)
@ -362,10 +485,36 @@ func (g *GitMode) PostProcess(md *data.ModeData) error {
 }

 func (g *GitMode) ImportName(pd *data.ProcessData, md *data.ModeData) string {
-	if misc.GetTagImportRegex(pd.ImportBranchPrefix, pd.AllowStreamBranches).MatchString(md.TagBranch) {
-		match := misc.GetTagImportRegex(pd.ImportBranchPrefix, pd.AllowStreamBranches).FindStringSubmatch(md.TagBranch)
+	if misc.GetTagImportRegex(pd).MatchString(md.TagBranch) {
+		match := misc.GetTagImportRegex(pd).FindStringSubmatch(md.TagBranch)
 		return match[3]
 	}

 	return strings.Replace(strings.TrimPrefix(md.TagBranch, "refs/heads/"), "%", "_", -1)
 }
+
+// Given a cdnUrl string as input, return same string, but with substituted
+// template values ( {{.Name}} , {{.Hash}}, {{.Filename}}, etc. )
+func ProcessUrl(cdnUrl string, name string, branch string, hash string, hashtype string, filename string) (string, bool) {
+	tmpUrl := Lookaside{name, branch, hash, hashtype, filename}
+
+	// Return cdnUrl as-is if we don't have any templates ("{{ .Variable }}") to process:
+	if !(strings.Contains(cdnUrl, "{{") && strings.Contains(cdnUrl, "}}")) {
+		return cdnUrl, false
+	}
+
+	// If we run into trouble with our template parsing, we'll just return the cdnUrl, exactly as we found it
+	tmpl, err := template.New("").Parse(cdnUrl)
+	if err != nil {
+		return cdnUrl, false
+	}
+
+	var result bytes.Buffer
+	err = tmpl.Execute(&result, tmpUrl)
+	if err != nil {
+		log.Fatalf("ERROR: Could not process CDN URL template(s) from URL string: %s\n", cdnUrl)
+	}
+
+	return result.String(), true
+
+}
--- a/pkg/rpmutils/regex.go
+++ b/pkg/rpmutils/regex.go
@ -2,10 +2,5 @@ package rpmutils

 import "regexp"

-var (
-	Nvr        = regexp.MustCompile("^(\\S+)-([\\w.]+)-(\\w+(?:\\.[\\w+]+)+?)(?:\\.(\\w+))?(?:\\.rpm)?$")
-	epoch      = regexp.MustCompile("(\\d+):")
-	module     = regexp.MustCompile("^(.+)-(.+)-([0-9]{19})\\.((?:.+){8})$")
-	dist       = regexp.MustCompile("(\\.el8(?:(?:_\\d)|))")
-	moduleDist = regexp.MustCompile("\\.module.+$")
-)
+// Nvr is a regular expression that matches a NVR.
+var Nvr = regexp.MustCompile("^(\\S+)-([\\w~%.+]+)-(\\w+(?:\\.[\\w~%+]+)+?)(?:\\.rpm)?$")
--- a/pkg/srpmproc/fetch.go
+++ b/pkg/srpmproc/fetch.go
@ -3,36 +3,41 @@ package srpmproc
 import (
 	"errors"
 	"fmt"
-	"github.com/rocky-linux/srpmproc/pkg/data"
 	"io"
 	"io/ioutil"
 	"log"
 	"net/http"
-	"os"
 	"path/filepath"
 	"strings"
+
+	"github.com/go-git/go-billy/v5"
+	"github.com/rocky-linux/srpmproc/pkg/blob"
+	"github.com/rocky-linux/srpmproc/pkg/data"
 )

-func Fetch(logger io.Writer, cdnUrl string, dir string) error {
+func Fetch(logger io.Writer, cdnUrl string, dir string, fs billy.Filesystem, storage blob.Storage) error {
 	pd := &data.ProcessData{
 		Log: log.New(logger, "", log.LstdFlags),
 	}

 	metadataPath := ""
-	err := filepath.Walk(dir, func(path string, info os.FileInfo, err error) error {
-		if strings.HasSuffix(path, ".metadata") {
-			if metadataPath != "" {
-				return errors.New("multiple metadata files")
-			}
-			metadataPath = path
-		}
-		return nil
-	})
+	ls, err := fs.ReadDir(dir)
 	if err != nil {
 		return err
 	}
+	for _, f := range ls {
+		if strings.HasSuffix(f.Name(), ".metadata") {
+			if metadataPath != "" {
+				return errors.New("multiple metadata files found")
+			}
+			metadataPath = filepath.Join(dir, f.Name())
+		}
+	}
+	if metadataPath == "" {
+		return errors.New("no metadata file found")
+	}

-	metadataFile, err := os.Open(metadataPath)
+	metadataFile, err := fs.Open(metadataPath)
 	if err != nil {
 		return fmt.Errorf("could not open metadata file: %v", err)
 	}
@ -53,31 +58,43 @@ func Fetch(logger io.Writer, cdnUrl string, dir string) error {
 			continue
 		}

-		lineInfo := strings.Split(line, " ")
-		hash := lineInfo[0]
-		path := lineInfo[1]
+		lineInfo := strings.SplitN(line, " ", 2)
+		hash := strings.TrimSpace(lineInfo[0])
+		path := strings.TrimSpace(lineInfo[1])

 		url := fmt.Sprintf("%s/%s", cdnUrl, hash)
+		if storage != nil {
+			url = hash
+		}
 		pd.Log.Printf("downloading %s", url)

-		req, err := http.NewRequest("GET", url, nil)
-		if err != nil {
-			return fmt.Errorf("could not create new http request: %v", err)
-		}
-		req.Header.Set("Accept-Encoding", "*")
+		var body []byte

-		resp, err := client.Do(req)
-		if err != nil {
-			return fmt.Errorf("could not download dist-git file: %v", err)
-		}
+		if storage != nil {
+			body, err = storage.Read(hash)
+			if err != nil {
+				return fmt.Errorf("could not read blob: %v", err)
+			}
+		} else {
+			req, err := http.NewRequest("GET", url, nil)
+			if err != nil {
+				return fmt.Errorf("could not create new http request: %v", err)
+			}
+			req.Header.Set("Accept-Encoding", "*")

-		body, err := ioutil.ReadAll(resp.Body)
-		if err != nil {
-			return fmt.Errorf("could not read the whole dist-git file: %v", err)
-		}
-		err = resp.Body.Close()
-		if err != nil {
-			return fmt.Errorf("could not close body handle: %v", err)
+			resp, err := client.Do(req)
+			if err != nil {
+				return fmt.Errorf("could not download dist-git file: %v", err)
+			}
+
+			body, err = ioutil.ReadAll(resp.Body)
+			if err != nil {
+				return fmt.Errorf("could not read the whole dist-git file: %v", err)
+			}
+			err = resp.Body.Close()
+			if err != nil {
+				return fmt.Errorf("could not close body handle: %v", err)
+			}
 		}

 		hasher := pd.CompareHash(body, hash)
@ -85,12 +102,12 @@ func Fetch(logger io.Writer, cdnUrl string, dir string) error {
 			return fmt.Errorf("checksum in metadata does not match dist-git file")
 		}

-		err = os.MkdirAll(filepath.Join(dir, filepath.Dir(path)), 0755)
+		err = fs.MkdirAll(filepath.Join(dir, filepath.Dir(path)), 0o755)
 		if err != nil {
-			return fmt.Errorf("could create all directories")
+			return fmt.Errorf("could not create all directories")
 		}

-		f, err := os.Create(filepath.Join(dir, path))
+		f, err := fs.Create(filepath.Join(dir, path))
 		if err != nil {
 			return fmt.Errorf("could not open file pointer: %v", err)
 		}
--- a/pkg/srpmproc/patch.go
+++ b/pkg/srpmproc/patch.go
@ -23,14 +23,17 @@ package srpmproc
 import (
 	"encoding/json"
 	"fmt"
-	"github.com/rocky-linux/srpmproc/pkg/misc"
-	"io/ioutil"
+	"io"
 	"log"
 	"os"
 	"path/filepath"
 	"strings"
 	"time"

+	"github.com/go-git/go-git/v5/plumbing/transport"
+	"github.com/rocky-linux/srpmproc/pkg/misc"
+
+	"github.com/go-git/go-billy/v5"
 	"github.com/go-git/go-billy/v5/memfs"
 	"github.com/go-git/go-git/v5"
 	"github.com/go-git/go-git/v5/config"
@ -45,10 +48,16 @@ import (

 func cfgPatches(pd *data.ProcessData, md *data.ModeData, patchTree *git.Worktree, pushTree *git.Worktree) error {
 	// check CFG patches
-	_, err := patchTree.Filesystem.Stat("ROCKY/CFG")
+	// use PATCHES directory if it exists otherwise ROCKY/CFG
+	cfgdir := "PATCHES"
+	_, err := patchTree.Filesystem.Stat(cfgdir)
+	if err != nil {
+		cfgdir = "ROCKY/CFG"
+	}
+	_, err = patchTree.Filesystem.Stat(cfgdir)
 	if err == nil {
 		// iterate through patches
-		infos, err := patchTree.Filesystem.ReadDir("ROCKY/CFG")
+		infos, err := patchTree.Filesystem.ReadDir(cfgdir)
 		if err != nil {
 			return fmt.Errorf("could not walk patches: %v", err)
 		}
@ -60,12 +69,12 @@ func cfgPatches(pd *data.ProcessData, md *data.ModeData, patchTree *git.Worktree
 			}

 			pd.Log.Printf("applying directive %s", info.Name())
-			filePath := filepath.Join("ROCKY/CFG", info.Name())
+			filePath := filepath.Join(cfgdir, info.Name())
 			directive, err := patchTree.Filesystem.Open(filePath)
 			if err != nil {
 				return fmt.Errorf("could not open directive file %s: %v", info.Name(), err)
 			}
-			directiveBytes, err := ioutil.ReadAll(directive)
+			directiveBytes, err := io.ReadAll(directive)
 			if err != nil {
 				return fmt.Errorf("could not read directive file: %v", err)
 			}
@ -93,7 +102,12 @@ func cfgPatches(pd *data.ProcessData, md *data.ModeData, patchTree *git.Worktree

 func applyPatches(pd *data.ProcessData, md *data.ModeData, patchTree *git.Worktree, pushTree *git.Worktree) error {
 	// check if patches exist
-	_, err := patchTree.Filesystem.Stat("ROCKY")
+	cfgdir := "PATCHES"
+	_, err := patchTree.Filesystem.Stat(cfgdir)
+	if err != nil {
+		cfgdir = "ROCKY"
+	}
+	_, err = patchTree.Filesystem.Stat(cfgdir)
 	if err == nil {
 		err := cfgPatches(pd, md, patchTree, pushTree)
 		if err != nil {
@ -141,37 +155,47 @@ func executePatchesRpm(pd *data.ProcessData, md *data.ModeData) error {
 	pd.Log.Printf("set reference to ref: %s", refName)

 	if err != nil {
-		// no patches active
-		log.Println("info: patch repo not found")
-		return nil
-	} else {
-		err = w.Checkout(&git.CheckoutOptions{
-			Branch: plumbing.NewRemoteReferenceName("origin", "main"),
-			Force:  true,
-		})
-		// common patches found, apply them
-		if err == nil {
-			err := applyPatches(pd, md, w, md.Worktree)
+		if err == transport.ErrInvalidAuthMethod || err == transport.ErrAuthenticationRequired {
+			fetchOptions.Auth = nil
+			err = repo.Fetch(fetchOptions)
 			if err != nil {
-				return err
+				// no patches active
+				log.Println("info: patch repo not found")
+				return nil
 			}
 		} else {
-			log.Println("info: no common patches found")
+			// no patches active
+			log.Println("info: patch repo not found")
+			return nil
 		}
+	}

-		err = w.Checkout(&git.CheckoutOptions{
-			Branch: plumbing.NewRemoteReferenceName("origin", md.PushBranch),
-			Force:  true,
-		})
-		// branch specific patches found, apply them
-		if err == nil {
-			err := applyPatches(pd, md, w, md.Worktree)
-			if err != nil {
-				return err
-			}
-		} else {
-			log.Println("info: no branch specific patches found")
+	err = w.Checkout(&git.CheckoutOptions{
+		Branch: plumbing.NewRemoteReferenceName("origin", "main"),
+		Force:  true,
+	})
+	// common patches found, apply them
+	if err == nil {
+		err := applyPatches(pd, md, w, md.Worktree)
+		if err != nil {
+			return err
 		}
+	} else {
+		log.Println("info: no common patches found")
+	}
+
+	err = w.Checkout(&git.CheckoutOptions{
+		Branch: plumbing.NewRemoteReferenceName("origin", md.PushBranch),
+		Force:  true,
+	})
+	// branch specific patches found, apply them
+	if err == nil {
+		err := applyPatches(pd, md, w, md.Worktree)
+		if err != nil {
+			return err
+		}
+	} else {
+		log.Println("info: no branch specific patches found")
 	}

 	return nil
@ -275,17 +299,34 @@ func patchModuleYaml(pd *data.ProcessData, md *data.ModeData) error {
 		return nil
 	}

-	mdTxtPath := "SOURCES/modulemd.src.txt"
-	f, err := md.Worktree.Filesystem.Open(mdTxtPath)
-	if err != nil {
-		mdTxtPath = "SOURCES/modulemd.txt"
+	mdTxtPath := ""
+	var f billy.File
+
+	// tagless mode implies we're looking for CentOS Stream modules, which are generally "SOURCES/NAME.yaml" (copied to SOURCES/ from import)
+	// if not tagless mode, proceed as usual with SOURCES/modulemd.*.txt
+	if pd.TaglessMode {
+		mdTxtPath = fmt.Sprintf("SOURCES/%s.yaml", md.Name)
 		f, err = md.Worktree.Filesystem.Open(mdTxtPath)
 		if err != nil {
-			return fmt.Errorf("could not open modulemd file: %v", err)
+			mdTxtPath = fmt.Sprintf("SOURCES/%s.yml", md.Name)
+			f, err = md.Worktree.Filesystem.Open(mdTxtPath)
+			if err != nil {
+				return fmt.Errorf("could not open modulemd file: %v", err)
+			}
+		}
+	} else {
+		mdTxtPath = "SOURCES/modulemd.src.txt"
+		f, err = md.Worktree.Filesystem.Open(mdTxtPath)
+		if err != nil {
+			mdTxtPath = "SOURCES/modulemd.txt"
+			f, err = md.Worktree.Filesystem.Open(mdTxtPath)
+			if err != nil {
+				return fmt.Errorf("could not open modulemd file: %v", err)
+			}
 		}
 	}

-	content, err := ioutil.ReadAll(f)
+	content, err := io.ReadAll(f)
 	if err != nil {
 		return fmt.Errorf("could not read modulemd file: %v", err)
 	}
@ -296,13 +337,30 @@ func patchModuleYaml(pd *data.ProcessData, md *data.ModeData) error {
 	}

 	// Get stream branch from tag
-	match := misc.GetTagImportRegex(pd.ImportBranchPrefix, pd.AllowStreamBranches).FindStringSubmatch(md.TagBranch)
-	streamBranch := strings.Split(match[2], "-")
-	// Force stream to be the same as stream name in branch
-	module.Data.Stream = streamBranch[len(streamBranch)-1]
+	// (in tagless mode we are trusting the "Stream: <VERSION>" text in the source YAML to be accurate)
+	if !pd.TaglessMode {
+		match := misc.GetTagImportRegex(pd).FindStringSubmatch(md.TagBranch)
+		streamBranch := strings.Split(match[2], "-")
+
+		// Force stream to be the same as stream name in branch
+		if module.V2 != nil {
+			module.V2.Data.Stream = streamBranch[len(streamBranch)-1]
+		}
+		if module.V3 != nil {
+			module.V3.Data.Stream = streamBranch[len(streamBranch)-1]
+		}
+	}
+
+	var components *modulemd.Components
+	if module.V2 != nil {
+		components = module.V2.Data.Components
+	}
+	if module.V3 != nil {
+		components = module.V3.Data.Components
+	}

 	log.Println("This module contains the following rpms:")
-	for name := range module.Data.Components.Rpms {
+	for name := range components.Rpms {
 		pd.Log.Printf("\t- %s", name)
 	}

@ -311,7 +369,7 @@ func patchModuleYaml(pd *data.ProcessData, md *data.ModeData) error {
 		defaultBranch = fmt.Sprintf("%s%d-stream-%s", pd.BranchPrefix, pd.Version, pd.ModuleFallbackStream)
 	}

-	for name, rpm := range module.Data.Components.Rpms {
+	for name, rpm := range components.Rpms {
 		var tipHash string
 		var pushBranch string

@ -350,6 +408,12 @@ func patchModuleYaml(pd *data.ProcessData, md *data.ModeData) error {
 			}
 		}

+		// If we got this far, we're good. This means the repos, branches, and commits exist.
+		// If this option is on, just use the branch name.
+		if pd.ModuleBranchNames {
+			tipHash = md.PushBranch
+		}
+
 		rpm.Ref = tipHash
 	}

--- a/pkg/srpmproc/process.go
+++ b/pkg/srpmproc/process.go
@ -21,9 +21,22 @@
 package srpmproc

 import (
+	"bufio"
 	"encoding/hex"
 	"fmt"
+	"io"
+	"io/ioutil"
+	"log"
+	"os"
+	"os/exec"
+	"os/user"
+	"path/filepath"
+	"strings"
+	"time"
+
 	"github.com/go-git/go-billy/v5"
+	"github.com/go-git/go-billy/v5/osfs"
+	"github.com/go-git/go-git/v5/plumbing/format/gitignore"
 	"github.com/go-git/go-git/v5/plumbing/transport"
 	"github.com/go-git/go-git/v5/plumbing/transport/http"
 	"github.com/go-git/go-git/v5/plumbing/transport/ssh"
@ -35,14 +48,6 @@ import (
 	"github.com/rocky-linux/srpmproc/pkg/misc"
 	"github.com/rocky-linux/srpmproc/pkg/modes"
 	"github.com/rocky-linux/srpmproc/pkg/rpmutils"
-	"io"
-	"io/ioutil"
-	"log"
-	"os"
-	"os/user"
-	"path/filepath"
-	"strings"
-	"time"

 	"github.com/go-git/go-billy/v5/memfs"
 	"github.com/go-git/go-git/v5"
@ -84,13 +89,27 @@ type ProcessDataRequest struct {
 	BranchPrefix         string
 	FsCreator            data.FsCreatorFunc
 	NoDupMode            bool
-	AllowStreamBranches  bool
+	BranchSuffix         string
+	StrictBranchMode     bool
 	ModuleFallbackStream string
 	NoStorageUpload      bool
 	NoStorageDownload    bool
 	SingleTag            string
 	CdnUrl               string
 	LogWriter            io.Writer
+
+	PackageVersion string
+	PackageRelease string
+
+	TaglessMode bool
+	Cdn         string
+
+	ModuleBranchNames bool
+}
+
+type LookasidePath struct {
+	Distro string
+	Url    string
 }

 func gitlabify(str string) string {
@ -101,7 +120,57 @@ func gitlabify(str string) string {
 	return strings.Replace(str, "+", "plus", -1)
 }

+// List of distros and their lookaside patterns
+// If we find one of these passed as --cdn (ex: "--cdn fedora"), then we override, and assign this URL to be our --cdn-url
+func StaticLookasides() []LookasidePath {
+	centos := LookasidePath{
+		Distro: "centos",
+		Url:    "https://git.centos.org/sources/{{.Name}}/{{.Branch}}/{{.Hash}}",
+	}
+	centosStream := LookasidePath{
+		Distro: "centos-stream",
+		Url:    "https://sources.stream.centos.org/sources/rpms/{{.Name}}/{{.Filename}}/{{.Hashtype}}/{{.Hash}}/{{.Filename}}",
+	}
+	rocky8 := LookasidePath{
+		Distro: "rocky8",
+		Url:    "https://rocky-linux-sources-staging.a1.rockylinux.org/{{.Hash}}",
+	}
+	rocky := LookasidePath{
+		Distro: "rocky",
+		Url:    "https://sources.build.resf.org/{{.Hash}}",
+	}
+	fedora := LookasidePath{
+		Distro: "fedora",
+		Url:    "https://src.fedoraproject.org/repo/pkgs/{{.Name}}/{{.Filename}}/{{.Hashtype}}/{{.Hash}}/{{.Filename}}",
+	}
+
+	return []LookasidePath{centos, centosStream, rocky8, rocky, fedora}
+
+}
+
+// Given a "--cdn" entry like "centos", we can search through our struct list of distros, and return the proper lookaside URL
+// If we can't find it, we return false and the calling function will error out
+func FindDistro(cdn string) (string, bool) {
+	var cdnUrl = ""
+
+	// Loop through each distro in the static list defined, try to find a match with "--cdn":
+	for _, distro := range StaticLookasides() {
+		if distro.Distro == strings.ToLower(cdn) {
+			cdnUrl = distro.Url
+			return cdnUrl, true
+		}
+	}
+	return "", false
+}
+
 func NewProcessData(req *ProcessDataRequest) (*data.ProcessData, error) {
+	// Build the logger to use for the data import
+	var writer io.Writer = os.Stdout
+	if req.LogWriter != nil {
+		writer = req.LogWriter
+	}
+	logger := log.New(writer, "", log.LstdFlags)
+
 	// Set defaults
 	if req.ModulePrefix == "" {
 		req.ModulePrefix = ModulePrefixCentOS
@ -131,6 +200,20 @@ func NewProcessData(req *ProcessDataRequest) (*data.ProcessData, error) {
 		req.CdnUrl = "https://git.centos.org/sources"
 	}

+	// If a Cdn distro is defined, we try to find a match from StaticLookasides() array of structs
+	// see if we have a match to --cdn (matching values are things like fedora, centos, rocky8, etc.)
+	// If we match, then we want to short-circuit the CdnUrl to the assigned distro's one
+	if req.Cdn != "" {
+		newCdn, foundDistro := FindDistro(req.Cdn)
+
+		if !foundDistro {
+			return nil, fmt.Errorf("Error, distro name given as --cdn argument is not valid.")
+		}
+
+		req.CdnUrl = newCdn
+		logger.Printf("Discovered --cdn distro: %s .  Using override CDN URL Pattern: %s", req.Cdn, req.CdnUrl)
+	}
+
 	// Validate required
 	if req.Package == "" {
 		return nil, fmt.Errorf("package cannot be empty")
@ -187,6 +270,9 @@ func NewProcessData(req *ProcessDataRequest) (*data.ProcessData, error) {
 	}

 	fsCreator := func(branch string) (billy.Filesystem, error) {
+		if req.TmpFsMode != "" {
+			return osfs.New(""), nil
+		}
 		return memfs.New(), nil
 	}
 	reqFsCreator := fsCreator
@ -194,12 +280,6 @@ func NewProcessData(req *ProcessDataRequest) (*data.ProcessData, error) {
 		reqFsCreator = req.FsCreator
 	}

-	var writer io.Writer = os.Stdout
-	if req.LogWriter != nil {
-		writer = req.LogWriter
-	}
-	logger := log.New(writer, "", log.LstdFlags)
-
 	if req.TmpFsMode != "" {
 		logger.Printf("using tmpfs dir: %s", req.TmpFsMode)
 		fsCreator = func(branch string) (billy.Filesystem, error) {
@ -208,7 +288,7 @@ func NewProcessData(req *ProcessDataRequest) (*data.ProcessData, error) {
 				return nil, err
 			}
 			tmpDir := filepath.Join(req.TmpFsMode, branch)
-			err = fs.MkdirAll(tmpDir, 0755)
+			err = fs.MkdirAll(tmpDir, 0o755)
 			if err != nil {
 				return nil, fmt.Errorf("could not create tmpfs dir: %v", err)
 			}
@ -248,10 +328,16 @@ func NewProcessData(req *ProcessDataRequest) (*data.ProcessData, error) {
 		NoStorageUpload:      req.NoStorageUpload,
 		ManualCommits:        manualCs,
 		ModuleFallbackStream: req.ModuleFallbackStream,
-		AllowStreamBranches:  req.AllowStreamBranches,
+		BranchSuffix:         req.BranchSuffix,
+		StrictBranchMode:     req.StrictBranchMode,
 		FsCreator:            fsCreator,
 		CdnUrl:               req.CdnUrl,
 		Log:                  logger,
+		PackageVersion:       req.PackageVersion,
+		PackageRelease:       req.PackageRelease,
+		TaglessMode:          req.TaglessMode,
+		Cdn:                  req.Cdn,
+		ModuleBranchNames:    req.ModuleBranchNames,
 	}, nil
 }

@ -263,6 +349,13 @@ func NewProcessData(req *ProcessDataRequest) (*data.ProcessData, error) {
 // all files that are remote goes into .gitignore
 // all ignored files' hash goes into .{Name}.metadata
 func ProcessRPM(pd *data.ProcessData) (*srpmprocpb.ProcessResponse, error) {
+	// if we are using "tagless mode", then we need to jump to a completely different import process:
+	// Version info needs to be derived from rpmbuild + spec file, not tags
+	if pd.TaglessMode {
+		result, err := processRPMTagless(pd)
+		return result, err
+	}
+
 	md, err := pd.Importer.RetrieveSource(pd)
 	if err != nil {
 		return nil, err
@ -302,6 +395,7 @@ func ProcessRPM(pd *data.ProcessData) (*srpmprocpb.ProcessResponse, error) {
 		list, err := remote.List(&git.ListOptions{
 			Auth: pd.Authenticator,
 		})
+
 		if err != nil {
 			log.Println("ignoring no-dup-mode")
 		} else {
@ -322,17 +416,17 @@ func ProcessRPM(pd *data.ProcessData) (*srpmprocpb.ProcessResponse, error) {
 	if pd.SingleTag != "" {
 		md.Branches = []string{fmt.Sprintf("refs/tags/%s", pd.SingleTag)}
 	} else if len(pd.ManualCommits) > 0 {
-		md.Branches = []string{}
-		for _, commit := range pd.ManualCommits {
-			branchCommit := strings.Split(commit, ":")
-			if len(branchCommit) != 2 {
-				return nil, fmt.Errorf("invalid manual commit list")
-			}
+		log.Println("Manual commits were listed for import.  Switching to perform a tagless import of these commit(s).")
+		pd.TaglessMode = true
+		return processRPMTagless(pd)
+	}

-			head := fmt.Sprintf("refs/tags/imports/%s/%s-%s", branchCommit[0], md.Name, branchCommit[1])
-			md.Branches = append(md.Branches, head)
-			commitPin[head] = branchCommit[1]
-		}
+	// If we have no valid branches to consider, then we'll automatically switch to attempt a tagless import:
+	if len(md.Branches) == 0 {
+		log.Println("No valid tags (refs/tags/imports/*) found in repository!  Switching to perform a tagless import.")
+		pd.TaglessMode = true
+		result, err := processRPMTagless(pd)
+		return result, err
 	}

 	for _, branch := range md.Branches {
@ -343,12 +437,8 @@ func ProcessRPM(pd *data.ProcessData) (*srpmprocpb.ProcessResponse, error) {
 			source.Expired = true
 		}

-		if strings.Contains(md.TagBranch, "-beta") {
-			continue
-		}
-
 		var matchString string
-		if !misc.GetTagImportRegex(pd.ImportBranchPrefix, pd.AllowStreamBranches).MatchString(md.TagBranch) {
+		if !misc.GetTagImportRegex(pd).MatchString(md.TagBranch) {
 			if pd.ModuleMode {
 				prefix := fmt.Sprintf("refs/heads/%s%d", pd.ImportBranchPrefix, pd.Version)
 				if strings.HasPrefix(md.TagBranch, prefix) {
@ -357,15 +447,17 @@ func ProcessRPM(pd *data.ProcessData) (*srpmprocpb.ProcessResponse, error) {
 					pd.Log.Printf("using match string: %s", matchString)
 				}
 			}
-			if !misc.GetTagImportRegex(pd.ImportBranchPrefix, pd.AllowStreamBranches).MatchString(matchString) {
+			if !misc.GetTagImportRegex(pd).MatchString(matchString) {
 				continue
 			}
 		} else {
 			matchString = md.TagBranch
 		}

-		match := misc.GetTagImportRegex(pd.ImportBranchPrefix, pd.AllowStreamBranches).FindStringSubmatch(matchString)
+		match := misc.GetTagImportRegex(pd).FindStringSubmatch(matchString)
+
 		md.PushBranch = pd.BranchPrefix + strings.TrimPrefix(match[2], pd.ImportBranchPrefix)
+
 		newTag := "imports/" + pd.BranchPrefix + strings.TrimPrefix(match[1], "imports/"+pd.ImportBranchPrefix)
 		newTag = strings.Replace(newTag, "%", "_", -1)

@ -465,7 +557,22 @@ func ProcessRPM(pd *data.ProcessData) (*srpmprocpb.ProcessResponse, error) {
 		}

 		// get ignored files hash and add to .{Name}.metadata
-		metadataFile := fmt.Sprintf(".%s.metadata", md.Name)
+		metadataFile := ""
+		ls, err := md.Worktree.Filesystem.ReadDir(".")
+		if err != nil {
+			return nil, fmt.Errorf("could not read directory: %v", err)
+		}
+		for _, f := range ls {
+			if strings.HasSuffix(f.Name(), ".metadata") {
+				if metadataFile != "" {
+					return nil, fmt.Errorf("multiple metadata files found")
+				}
+				metadataFile = f.Name()
+			}
+		}
+		if metadataFile == "" {
+			metadataFile = fmt.Sprintf(".%s.metadata", md.Name)
+		}
 		metadata, err := w.Filesystem.Create(metadataFile)
 		if err != nil {
 			return nil, fmt.Errorf("could not create metadata file: %v", err)
@ -482,7 +589,7 @@ func ProcessRPM(pd *data.ProcessData) (*srpmprocpb.ProcessResponse, error) {
 			if err != nil {
 				return nil, fmt.Errorf("could not open ignored source file %s: %v", sourcePath, err)
 			}
-			sourceFileBts, err := ioutil.ReadAll(sourceFile)
+			sourceFileBts, err := io.ReadAll(sourceFile)
 			if err != nil {
 				return nil, fmt.Errorf("could not read the whole of ignored source file: %v", err)
 			}
@ -551,6 +658,12 @@ func ProcessRPM(pd *data.ProcessData) (*srpmprocpb.ProcessResponse, error) {

 		// show status
 		status, _ := w.Status()
+		if !pd.ModuleMode {
+			if status.IsClean() {
+				pd.Log.Printf("No changes detected. Our downstream is up to date.")
+				continue
+			}
+		}
 		pd.Log.Printf("successfully processed:\n%s", status)

 		statusLines := strings.Split(status.String(), "\n")
@ -634,3 +747,631 @@ func ProcessRPM(pd *data.ProcessData) (*srpmprocpb.ProcessResponse, error) {
 		BranchVersions: versionForBranch,
 	}, nil
 }
+
+// Process for when we want to import a tagless repo (like from CentOS Stream)
+func processRPMTagless(pd *data.ProcessData) (*srpmprocpb.ProcessResponse, error) {
+	pd.Log.Println("Tagless mode detected, attempting import of latest commit")
+
+	// In tagless mode, we *automatically* set StrictBranchMode to true
+	// Only the exact <PREFIX><VERSION><SUFFIX> branch should be pulled from the source repo
+	pd.StrictBranchMode = true
+
+	// our return values: a mapping of branches -> commits (1:1) that we're bringing in,
+	// and a mapping of branches to: version = X, release = Y
+	latestHashForBranch := map[string]string{}
+	versionForBranch := map[string]*srpmprocpb.VersionRelease{}
+
+	md, err := pd.Importer.RetrieveSource(pd)
+	if err != nil {
+		pd.Log.Println("Error detected in  RetrieveSource!")
+		return nil, err
+	}
+
+	md.BlobCache = map[string][]byte{}
+
+	// TODO: add tagless module support
+	remotePrefix := "rpms"
+	if pd.ModuleMode {
+		remotePrefix = "modules"
+	}
+
+	// Set up our remote URL for pushing our repo to
+	var tagIgnoreList []string
+	if pd.NoDupMode {
+		repo, err := git.Init(memory.NewStorage(), memfs.New())
+		if err != nil {
+			return nil, fmt.Errorf("could not init git repo: %v", err)
+		}
+		remoteUrl := fmt.Sprintf("%s/%s/%s.git", pd.UpstreamPrefix, remotePrefix, gitlabify(md.Name))
+		refspec := config.RefSpec("+refs/heads/*:refs/remotes/origin/*")
+
+		remote, err := repo.CreateRemote(&config.RemoteConfig{
+			Name:  "origin",
+			URLs:  []string{remoteUrl},
+			Fetch: []config.RefSpec{refspec},
+		})
+		if err != nil {
+			return nil, fmt.Errorf("could not create remote: %v", err)
+		}
+
+		list, err := remote.List(&git.ListOptions{
+			Auth: pd.Authenticator,
+		})
+		if err != nil {
+			log.Println("ignoring no-dup-mode")
+		} else {
+			for _, ref := range list {
+				if !strings.HasPrefix(string(ref.Name()), "refs/tags/imports") {
+					continue
+				}
+				tagIgnoreList = append(tagIgnoreList, string(ref.Name()))
+			}
+		}
+	}
+
+	sourceRepo := *md.Repo
+	sourceWorktree := *md.Worktree
+	localPath := ""
+
+	// if a manual commit list is provided, we want to create our md.Branches[] array in a special format:
+	if len(pd.ManualCommits) > 0 {
+		md.Branches = []string{}
+		for _, commit := range pd.ManualCommits {
+			branchCommit := strings.Split(commit, ":")
+			if len(branchCommit) != 2 {
+				return nil, fmt.Errorf("invalid manual commit list")
+			}
+
+			head := fmt.Sprintf("COMMIT:%s:%s", branchCommit[0], branchCommit[1])
+			md.Branches = append(md.Branches, head)
+		}
+	}
+
+	for _, branch := range md.Branches {
+		md.Repo = &sourceRepo
+		md.Worktree = &sourceWorktree
+		md.TagBranch = branch
+
+		for _, source := range md.SourcesToIgnore {
+			source.Expired = true
+		}
+
+		// Create a temporary place to check out our tag/branch : /tmp/srpmproctmp_<PKG_NAME><RANDOMSTRING>/
+		localPath, _ = os.MkdirTemp("/tmp", fmt.Sprintf("srpmproctmp_%s", md.Name))
+
+		if err := os.RemoveAll(localPath); err != nil {
+			return nil, fmt.Errorf("Could not remove previous temporary directory: %s", localPath)
+		}
+		if err := os.Mkdir(localPath, 0o755); err != nil {
+			return nil, fmt.Errorf("Could not create temporary directory: %s", localPath)
+		}
+
+		// we'll make our branch we're processing more presentable if it's in the COMMIT:<branch>:<hash> format:
+		if strings.HasPrefix(branch, "COMMIT:") {
+			branch = fmt.Sprintf("refs/heads/%s", strings.Split(branch, ":")[1])
+		}
+
+		// Clone repo into the temporary path, but only the tag we're interested in:
+		// (TODO: will probably need to assign this a variable or use the md struct gitrepo object to perform a successful tag+push later)
+		rTmp, err := git.PlainClone(localPath, false, &git.CloneOptions{
+			URL:           pd.RpmLocation,
+			SingleBranch:  true,
+			ReferenceName: plumbing.ReferenceName(branch),
+		})
+		if err != nil {
+			return nil, err
+		}
+
+		// If we're dealing with a special manual commit to import ("COMMIT:<branch>:<githash>"), then we need to check out that
+		// specific hash from the repo we are importing from:
+		if strings.HasPrefix(md.TagBranch, "COMMIT:") {
+			commitList := strings.Split(md.TagBranch, ":")
+
+			wTmp, _ := rTmp.Worktree()
+			err = wTmp.Checkout(&git.CheckoutOptions{
+				Hash: plumbing.NewHash(commitList[2]),
+			})
+			if err != nil {
+				return nil, fmt.Errorf("Could not find manual commit %s in the repository.  Must be a valid commit hash.", commitList[2])
+			}
+		}
+
+		// Now that we're cloned into localPath, we need to "covert" the import into the old format
+		// We want sources to become .PKGNAME.metadata, we want SOURCES and SPECS folders, etc.
+		repoFixed, _ := convertLocalRepo(md.Name, localPath)
+		if !repoFixed {
+			return nil, fmt.Errorf("Error converting repository into SOURCES + SPECS + .package.metadata format")
+		}
+
+		// call extra function to determine the proper way to convert the tagless branch name.
+		// c9s becomes r9s (in the usual case), or in the modular case, stream-httpd-2.4-rhel-9.1.0 becomes r9s-stream-httpd-2.4_r9.1.0
+		md.PushBranch = taglessBranchName(branch, pd)
+
+		rpmVersion := ""
+
+		// get name-version-release of tagless repo, only if we're not a module repo:
+		if !pd.ModuleMode {
+			nvrString, err := getVersionFromSpec(localPath, pd.Version)
+			if err != nil {
+				return nil, err
+			}
+
+			// Set version and release fields we extracted (name|version|release are separated by pipes)
+			pd.PackageVersion = strings.Split(nvrString, "|")[1]
+			pd.PackageRelease = strings.Split(nvrString, "|")[2]
+
+			// Set full rpm version:  name-version-release (for tagging properly)
+			rpmVersion = fmt.Sprintf("%s-%s-%s", md.Name, pd.PackageVersion, pd.PackageRelease)
+
+			pd.Log.Println("Successfully determined version of tagless checkout: ", rpmVersion)
+		} else {
+			// In case of module mode, we just set rpmVersion to the current date - that's what our tag will end up being
+			rpmVersion = time.Now().Format("2006-01-02")
+		}
+
+		// Make an initial repo we will use to push to our target
+		pushRepo, err := git.PlainInit(localPath+"_gitpush", false)
+		if err != nil {
+			return nil, fmt.Errorf("could not create new dist Repo: %v", err)
+		}
+
+		w, err := pushRepo.Worktree()
+		if err != nil {
+			return nil, fmt.Errorf("could not get dist Worktree: %v", err)
+		}
+
+		// Create a remote "origin" in our empty git, make the upstream equal to the branch we want to modify
+		pushUrl := fmt.Sprintf("%s/%s/%s.git", pd.UpstreamPrefix, remotePrefix, gitlabify(md.Name))
+		refspec := config.RefSpec(fmt.Sprintf("+refs/heads/%s:refs/remotes/origin/%s", md.PushBranch, md.PushBranch))
+
+		// Make our remote repo the target one - the one we want to push our update to
+		pushRepoRemote, err := pushRepo.CreateRemote(&config.RemoteConfig{
+			Name:  "origin",
+			URLs:  []string{pushUrl},
+			Fetch: []config.RefSpec{refspec},
+		})
+		if err != nil {
+			return nil, fmt.Errorf("could not create remote: %v", err)
+		}
+
+		// fetch our branch data (md.PushBranch) into this new repo
+		err = pushRepo.Fetch(&git.FetchOptions{
+			RemoteName: "origin",
+			RefSpecs:   []config.RefSpec{refspec},
+			Auth:       pd.Authenticator,
+		})
+
+		refName := plumbing.NewBranchReferenceName(md.PushBranch)
+
+		var hash plumbing.Hash
+		h := plumbing.NewSymbolicReference(plumbing.HEAD, refName)
+		if err := pushRepo.Storer.CheckAndSetReference(h, nil); err != nil {
+			return nil, fmt.Errorf("Could not set symbolic reference: %v", err)
+		}
+
+		err = w.Checkout(&git.CheckoutOptions{
+			Branch: plumbing.NewRemoteReferenceName("origin", md.PushBranch),
+			Hash:   hash,
+			Force:  true,
+		})
+
+		// These os commands actually move the data from our cloned import repo to our cloned "push" repo (upstream -> downstream)
+		// First we clobber the target push repo's data, and then move the new data into it from the upstream repo we cloned earlier
+		os.RemoveAll(fmt.Sprintf("%s_gitpush/SPECS", localPath))
+		os.RemoveAll(fmt.Sprintf("%s_gitpush/SOURCES", localPath))
+		os.RemoveAll(fmt.Sprintf("%s_gitpush/.gitignore", localPath))
+		os.RemoveAll(fmt.Sprintf("%s_gitpush/%s.metadata", localPath, md.Name))
+		os.Rename(fmt.Sprintf("%s/SPECS", localPath), fmt.Sprintf("%s_gitpush/SPECS", localPath))
+		os.Rename(fmt.Sprintf("%s/SOURCES", localPath), fmt.Sprintf("%s_gitpush/SOURCES", localPath))
+		os.Rename(fmt.Sprintf("%s/.gitignore", localPath), fmt.Sprintf("%s_gitpush/.gitignore", localPath))
+		os.Rename(fmt.Sprintf("%s/.%s.metadata", localPath, md.Name), fmt.Sprintf("%s_gitpush/.%s.metadata", localPath, md.Name))
+
+		md.Repo = pushRepo
+		md.Worktree = w
+
+		// Download lookaside sources (tarballs) into the push git repo:
+		err = pd.Importer.WriteSource(pd, md)
+		if err != nil {
+			return nil, err
+		}
+
+		// Call function to upload source to target lookaside and
+		// ensure the sources are added to .gitignore
+		err = processLookasideSources(pd, md, localPath+"_gitpush")
+		if err != nil {
+			return nil, err
+		}
+
+		// Apply patch(es) if needed:
+		if pd.ModuleMode {
+			err := patchModuleYaml(pd, md)
+			if err != nil {
+				return nil, err
+			}
+		} else {
+			err := executePatchesRpm(pd, md)
+			if err != nil {
+				return nil, err
+			}
+		}
+
+		err = w.AddWithOptions(&git.AddOptions{All: true})
+		if err != nil {
+			return nil, fmt.Errorf("Error adding SOURCES/ , SPECS/ or .metadata file to commit list.")
+		}
+
+		status, err := w.Status()
+		if !pd.ModuleMode {
+			if status.IsClean() {
+				pd.Log.Printf("No changes detected. Our downstream is up to date.")
+				continue
+			}
+		}
+		pd.Log.Printf("successfully processed:\n%s", status)
+
+		// assign tag for our new remote we're about to push (derived from the SRPM version)
+		newTag := "refs/tags/imports/" + md.PushBranch + "/" + rpmVersion
+		newTag = strings.Replace(newTag, "%", "_", -1)
+
+		// pushRefspecs is a list of all the references we want to push (tags + heads)
+		// It's an array of colon-separated strings which map local references to their remote counterparts
+		var pushRefspecs []config.RefSpec
+
+		// We need to find out if the remote repo already has this branch
+		// If it doesn't, we want to add *:* to our references for commit.  This will allow us to push the new branch
+		// If it does, we can simply push HEAD:refs/heads/<BRANCH>
+		newRepo := true
+		refList, _ := pushRepoRemote.List(&git.ListOptions{Auth: pd.Authenticator})
+		for _, ref := range refList {
+			if strings.HasSuffix(ref.Name().String(), fmt.Sprintf("heads/%s", md.PushBranch)) {
+				newRepo = false
+				break
+			}
+		}
+
+		if newRepo {
+			pushRefspecs = append(pushRefspecs, config.RefSpec("*:*"))
+			pd.Log.Printf("New remote repo detected, creating new remote branch")
+		}
+
+		// Identify specific references we want to push
+		// Should be refs/heads/<target_branch>, and a tag called imports/<target_branch>/<rpm_nvr>
+		pushRefspecs = append(pushRefspecs, config.RefSpec(fmt.Sprintf("HEAD:refs/heads/%s", md.PushBranch)))
+		pushRefspecs = append(pushRefspecs, config.RefSpec(fmt.Sprintf("HEAD:%s", newTag)))
+
+		// Actually do the commit (locally)
+		commit, err := w.Commit("import from tagless source "+pd.Importer.ImportName(pd, md), &git.CommitOptions{
+			Author: &object.Signature{
+				Name:  pd.GitCommitterName,
+				Email: pd.GitCommitterEmail,
+				When:  time.Now(),
+			},
+		})
+		if err != nil {
+			return nil, fmt.Errorf("could not commit object: %v", err)
+		}
+
+		obj, err := pushRepo.CommitObject(commit)
+		if err != nil {
+			return nil, fmt.Errorf("could not get commit object: %v", err)
+		}
+
+		pd.Log.Printf("Committed local repo tagless mode transform:\n%s", obj.String())
+
+		// After commit, we will now tag our local repo on disk:
+		_, err = pushRepo.CreateTag(newTag, commit, &git.CreateTagOptions{
+			Tagger: &object.Signature{
+				Name:  pd.GitCommitterName,
+				Email: pd.GitCommitterEmail,
+				When:  time.Now(),
+			},
+			Message: "import " + md.TagBranch + " from " + pd.RpmLocation + "(import from tagless source)",
+			SignKey: nil,
+		})
+		if err != nil {
+			return nil, fmt.Errorf("could not create tag: %v", err)
+		}
+
+		pd.Log.Printf("Pushing these references to the remote:  %+v \n", pushRefspecs)
+
+		// Do the actual push to the remote target repository
+		err = pushRepo.Push(&git.PushOptions{
+			RemoteName: "origin",
+			Auth:       pd.Authenticator,
+			RefSpecs:   pushRefspecs,
+			Force:      true,
+		})
+
+		if err != nil {
+			return nil, fmt.Errorf("could not push to remote: %v", err)
+		}
+
+		if err := os.RemoveAll(localPath); err != nil {
+			log.Printf("Error cleaning up temporary git checkout directory %s .  Non-fatal, continuing anyway...\n", localPath)
+		}
+		if err := os.RemoveAll(fmt.Sprintf("%s_gitpush", localPath)); err != nil {
+			log.Printf("Error cleaning up temporary git checkout directory %s .  Non-fatal, continuing anyway...\n", fmt.Sprintf("%s_gitpush", localPath))
+		}
+
+		// append our processed branch to the return structures:
+		latestHashForBranch[md.PushBranch] = obj.Hash.String()
+
+		versionForBranch[md.PushBranch] = &srpmprocpb.VersionRelease{
+			Version: pd.PackageVersion,
+			Release: pd.PackageRelease,
+		}
+	}
+
+	// return struct with all our branch:commit and branch:version+release mappings
+	return &srpmprocpb.ProcessResponse{
+		BranchCommits:  latestHashForBranch,
+		BranchVersions: versionForBranch,
+	}, nil
+}
+
+// Given a local repo on disk, ensure it's in the "traditional" format.  This means:
+//   - metadata file is named .pkgname.metadata
+//   - metadata file has the old "<SHASUM>  SOURCES/<filename>"  format
+//   - SPECS/ and SOURCES/ exist and are populated correctly
+func convertLocalRepo(pkgName string, localRepo string) (bool, error) {
+	// Make sure we have a SPECS and SOURCES folder made:
+	if err := os.MkdirAll(fmt.Sprintf("%s/SOURCES", localRepo), 0o755); err != nil {
+		return false, fmt.Errorf("Could not create SOURCES directory in: %s", localRepo)
+	}
+
+	if err := os.MkdirAll(fmt.Sprintf("%s/SPECS", localRepo), 0o755); err != nil {
+		return false, fmt.Errorf("Could not create SPECS directory in: %s", localRepo)
+	}
+
+	// Loop through each file/folder and operate accordingly:
+	files, err := ioutil.ReadDir(localRepo)
+	if err != nil {
+		return false, err
+	}
+
+	for _, f := range files {
+		// We don't want to process SOURCES, SPECS, or any of our .git folders
+		if f.Name() == "SOURCES" || f.Name() == "SPECS" || strings.HasPrefix(f.Name(), ".git") || f.Name() == "."+pkgName+".metadata" {
+			continue
+		}
+
+		// If we have a metadata "sources" file, we need to read it and convert to the old .<pkgname>.metadata format
+		if f.Name() == "sources" {
+			convertStatus := convertMetaData(pkgName, localRepo)
+
+			if convertStatus != true {
+				return false, fmt.Errorf("Error converting sources metadata file to .metadata format")
+			}
+
+			continue
+		}
+
+		// Any file that ends in a ".spec" should be put into SPECS/
+		if strings.HasSuffix(f.Name(), ".spec") {
+			err := os.Rename(fmt.Sprintf("%s/%s", localRepo, f.Name()), fmt.Sprintf("%s/SPECS/%s", localRepo, f.Name()))
+			if err != nil {
+				return false, fmt.Errorf("Error moving .spec file to SPECS/")
+			}
+		}
+
+		// if a file isn't skipped in one of the above checks, then it must be a file that belongs in SOURCES/
+		os.Rename(fmt.Sprintf("%s/%s", localRepo, f.Name()), fmt.Sprintf("%s/SOURCES/%s", localRepo, f.Name()))
+	}
+
+	return true, nil
+}
+
+// Given a local "sources" metadata file (new CentOS Stream format), convert it into the older
+// classic CentOS style:  "<HASH>  SOURCES/<FILENAME>"
+func convertMetaData(pkgName string, localRepo string) bool {
+	lookAside, err := os.Open(fmt.Sprintf("%s/sources", localRepo))
+	if err != nil {
+		return false
+	}
+
+	// Split file into lines and start processing:
+	scanner := bufio.NewScanner(lookAside)
+	scanner.Split(bufio.ScanLines)
+
+	// convertedLA is our array of new "converted" lookaside lines
+	var convertedLA []string
+
+	// loop through each line, and:
+	//   - split by whitespace
+	//   - check each line begins with "SHA" or "MD" - validate
+	//   - take the
+	// Then check
+	for scanner.Scan() {
+		tmpLine := strings.Fields(scanner.Text())
+		// make sure line starts with a "SHA" or "MD" before processing - otherwise it might not be a valid format lookaside line!
+		if !(strings.HasPrefix(tmpLine[0], "SHA") || strings.HasPrefix(tmpLine[0], "MD")) {
+			continue
+		}
+
+		// Strip out "( )" characters from file name and prepend SOURCES/ to it
+		tmpLine[1] = strings.ReplaceAll(tmpLine[1], "(", "")
+		tmpLine[1] = strings.ReplaceAll(tmpLine[1], ")", "")
+		tmpLine[1] = fmt.Sprintf("SOURCES/%s", tmpLine[1])
+
+		convertedLA = append(convertedLA, fmt.Sprintf("%s %s", tmpLine[3], tmpLine[1]))
+	}
+	lookAside.Close()
+
+	// open .<NAME>.metadata file for writing our old-format lines
+	lookAside, err = os.OpenFile(fmt.Sprintf("%s/.%s.metadata", localRepo, pkgName), os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644)
+	if err != nil {
+		fmt.Errorf("Error opening new .metadata file for writing.")
+		return false
+	}
+
+	writer := bufio.NewWriter(lookAside)
+
+	for _, convertedLine := range convertedLA {
+		_, _ = writer.WriteString(convertedLine + "\n")
+	}
+
+	writer.Flush()
+	lookAside.Close()
+
+	// Remove old "sources" metadata file - we don't need it now that conversion is complete
+	os.Remove(fmt.Sprintf("%s/sources", localRepo))
+
+	return true
+}
+
+// Given a local checked out folder and package name, including SPECS/ , SOURCES/ , and .package.metadata, this will:
+//   - create a "dummy" SRPM (using dummy sources files we use to populate tarballs from lookaside)
+//   - extract RPM version info from that SRPM, and return it
+//
+// If we are in tagless mode, we need to get a package version somehow!
+func getVersionFromSpec(localRepo string, majorVersion int) (string, error) {
+	// Make sure we have "rpm" and "rpmbuild" and "cp" available in our PATH.  Otherwise, this won't work:
+	_, err := exec.LookPath("rpmspec")
+	if err != nil {
+		return "", fmt.Errorf("Could not find rpmspec program in PATH")
+	}
+
+	// Read the first file from SPECS/ to get our spec file
+	// (there should only be one file - we check that it ends in ".spec" just to be sure!)
+	lsTmp, err := ioutil.ReadDir(fmt.Sprintf("%s/SPECS/", localRepo))
+	if err != nil {
+		return "", err
+	}
+	specFile := lsTmp[0].Name()
+
+	if !strings.HasSuffix(specFile, ".spec") {
+		return "", fmt.Errorf("First file found in SPECS/ is not a .spec file!  Check the SPECS/ directory in the repo?")
+	}
+
+	// Call the rpmspec binary to extract the version-release info out of it, and tack on ".el<VERSION>" at the end:
+	cmdArgs := []string{
+		"--srpm",
+		fmt.Sprintf(`--define=dist  .el%d`, majorVersion),
+		fmt.Sprintf(`--define=_topdir  %s`, localRepo),
+		"-q",
+		"--queryformat",
+		`%{NAME}|%{VERSION}|%{RELEASE}\n`,
+		fmt.Sprintf("%s/SPECS/%s", localRepo, specFile),
+	}
+	cmd := exec.Command("rpmspec", cmdArgs...)
+	nvrTmp, err := cmd.Output()
+	if err != nil {
+		return "", fmt.Errorf("Error running rpmspec command to determine RPM name-version-release identifier. \nCommand attempted: %s \nCommand output: %s", cmd.String(), string(nvrTmp))
+	}
+
+	// Pull first line of the version output to get the name-version-release number (there should only be 1 line)
+	nvr := string(nvrTmp)
+	nvr = strings.Fields(nvr)[0]
+
+	// return name-version-release string we derived:
+	log.Printf("Derived NVR %s from tagless repo via rpmspec command\n", nvr)
+	return nvr, nil
+}
+
+// We need to loop through the lookaside blob files ("SourcesToIgnore"),
+// and upload them to our target storage (usually an S3 bucket, but could be a local folder)
+//
+// We also need to add the source paths to .gitignore in the git repo, so we don't accidentally commit + push them
+func processLookasideSources(pd *data.ProcessData, md *data.ModeData, localDir string) error {
+	w := md.Worktree
+	metadata, err := w.Filesystem.Create(fmt.Sprintf(".%s.metadata", md.Name))
+	if err != nil {
+		return fmt.Errorf("could not create metadata file: %v", err)
+	}
+
+	// Keep track of files we've already uploaded - don't want duplicates!
+	var alreadyUploadedBlobs []string
+
+	gitIgnore, err := os.OpenFile(fmt.Sprintf("%s/.gitignore", localDir), os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644)
+	if err != nil {
+		return err
+	}
+
+	for _, source := range md.SourcesToIgnore {
+
+		sourcePath := source.Name
+		_, err := w.Filesystem.Stat(sourcePath)
+		if source.Expired || err != nil {
+			continue
+		}
+
+		sourceFile, err := w.Filesystem.Open(sourcePath)
+		if err != nil {
+			return fmt.Errorf("could not open ignored source file %s: %v", sourcePath, err)
+		}
+		sourceFileBts, err := io.ReadAll(sourceFile)
+		if err != nil {
+			return fmt.Errorf("could not read the whole of ignored source file: %v", err)
+		}
+
+		source.HashFunction.Reset()
+		_, err = source.HashFunction.Write(sourceFileBts)
+		if err != nil {
+			return fmt.Errorf("could not write bytes to hash function: %v", err)
+		}
+		checksum := hex.EncodeToString(source.HashFunction.Sum(nil))
+		checksumLine := fmt.Sprintf("%s %s\n", checksum, sourcePath)
+		_, err = metadata.Write([]byte(checksumLine))
+		if err != nil {
+			return fmt.Errorf("could not write to metadata file: %v", err)
+		}
+
+		if data.StrContains(alreadyUploadedBlobs, checksum) {
+			continue
+		}
+		exists, err := pd.BlobStorage.Exists(checksum)
+		if err != nil {
+			return err
+		}
+		if !exists && !pd.NoStorageUpload {
+			err := pd.BlobStorage.Write(checksum, sourceFileBts)
+			if err != nil {
+				return err
+			}
+			pd.Log.Printf("wrote %s to blob storage", checksum)
+		}
+		alreadyUploadedBlobs = append(alreadyUploadedBlobs, checksum)
+
+		// Add this SOURCES/ lookaside file to be excluded
+		w.Excludes = append(w.Excludes, gitignore.ParsePattern(sourcePath, nil))
+
+		// Append the SOURCES/<file> path to .gitignore:
+		_, err = gitIgnore.Write([]byte(fmt.Sprintf("%s\n", sourcePath)))
+		if err != nil {
+			return err
+		}
+
+	}
+
+	err = gitIgnore.Close()
+	if err != nil {
+		return err
+	}
+
+	return nil
+}
+
+// Given an input branch name to import from, like "refs/heads/c9s", produce the tagless branch name we want to commit to, like "r9s"
+// Modular translation of CentOS stream branches i is also done - branch stream-maven-3.8-rhel-9.1.0  ---->  r9s-stream-maven-3.8_9.1.0
+func taglessBranchName(fullBranch string, pd *data.ProcessData) string {
+	// Split the full branch name "refs/heads/blah" to only get the short name - last entry
+	tmpBranch := strings.Split(fullBranch, "/")
+	branch := tmpBranch[len(tmpBranch)-1]
+
+	// Simple case:  if our branch is not a modular stream branch, just return the normal <prefix><version><suffix> pattern
+	if !strings.HasPrefix(branch, "stream-") {
+		return fmt.Sprintf("%s%d%s", pd.BranchPrefix, pd.Version, pd.BranchSuffix)
+	}
+
+	// index where the "-rhel-" starts near the end of the string
+	rhelSpot := strings.LastIndex(branch, "-rhel-")
+
+	// module name will be everything from the start until that "-rhel-" string (like "stream-httpd-2.4")
+	moduleString := branch[0:rhelSpot]
+
+	// major minor version is everything after the "-rhel-" string
+	majorMinor := branch[rhelSpot+6:]
+
+	// return translated modular branch:
+	return fmt.Sprintf("%s%d%s-%s_%s", pd.BranchPrefix, pd.Version, pd.BranchSuffix, moduleString, majorMinor)
+}
Author	SHA1	Message	Date
Mustafa Gezen	f2864a875a	Merge pull request #16 from dirkmueller/openela Support the openELA "PATCHES" directory format as well	2023-11-16 18:04:50 +01:00
Dirk Müller	82271f5d9e	Support the openELA "PATCHES" directory format as well	2023-11-06 16:53:43 +01:00
Mustafa Gezen	963b5d25dd	Fix goreleaser archive name template	2023-10-06 08:13:19 +02:00
Mustafa Gezen	a5f9082c15	Be compatible with latest goreleaser	2023-10-06 08:05:29 +02:00
Mustafa Gezen	f2374845e3	Merge pull request #15 from nazunalika/main	2023-10-06 07:56:14 +02:00
Louis Abel	162e7cea34	continue as it's a loop	2023-09-19 17:16:08 -07:00
Louis Abel	edeb0239df	Add two features and one fix This commit attempts to merge in custom changes on distrobuild. In particular: * Imports * Package: If there are no changes, do not import. This currently returns as an error, so may need adjusting. * Module: Import, regardless if there is changes or not. MBS cares more about the fact there's a commit associated with a module build. It will not build off the same commit ID. * Add --module-branch-names-only flag to make it so module yamls will have a branch name rather than commit ID. This makes it easier to work with MBS. * Fixes * tagless imports should use the _topdir macro, otherwise a bug may surface where macros or other data cannot be determined, and a cryptic error may display.	2023-09-05 03:50:43 -07:00
Mustafa Gezen	40180a342b	Merge pull request #11 from skip77/manual-commits-fix Manual commits fix	2023-03-12 03:06:01 +01:00
skip77	7f14563f93	Merge branch 'rocky-linux:main' into manual-commits-fix	2023-03-11 17:38:18 -05:00
Mustafa Gezen	6e2ed2eb23	Merge pull request #8 from skip77/main	2023-03-11 23:26:54 +01:00
Skip Grube	44725a727d	Repaired manual commits feature and added some bugfixes: - Manual commits work properly again, and are passed to "tagless mode" for version discovery + tagging - Fixed tagless mode issue with rpmspec warnings, only stdout is considered - Fixed tagless mode issue where renamed folders were not properly overwritten, causing potential bad commits -Skip G.	2023-03-05 22:04:20 -05:00
Skip Grube	4f975913b4	Removed old commented line -Skip	2023-02-28 16:23:51 -05:00
Skip Grube	3371826851	Fixed up formatting issues from rpmspec commit -Skip G.	2023-02-28 15:25:11 -05:00
skip77	f4f2a0a9e4	Merge branch 'rocky-linux:main' into main	2023-02-27 16:32:24 -05:00
skip77	5e8ed2e521	Fix Default cdn behavior (#9 ) - CDN returned as-is for processing if no template indicators are found -Skip G.	2023-02-27 15:26:21 -05:00
Skip Grube	2eb0768f33	Switched to using rpmspec for version determination in tagless mode: - Greatly simplified getVersionFromSpec function due to rpmspec usage - Function now returns error info in case rpmspec shell command goes wrong -Skip Grube	2023-02-26 22:59:24 -05:00
Mustafa Gezen	2c6d6f0b0a	Merge pull request #7 from skip77/main	2023-02-24 02:16:54 +01:00
Skip Grube	0ea6123b6e	More cleanup from feedback - Removed extra lines - Externalized Lookaside struct{} -Skip Grube	2023-02-23 13:57:21 -05:00
Skip Grube	2cc3bcc3a3	Updated cdn-related commits for style and cleanup -Skip Grube	2023-02-23 12:31:31 -05:00
Skip Grube	3237794071	Updates to add flexibility to lookasides and tagless handling: - Added template "macros" to allow for complex --cdn-url specifications ( {{.Name}}, {{.Branch}}, etc.) - Added --cdn <distro> option with pre-set URL patterns to simplify use - Kept default fallback behavior and search patterns of --cdn-url if templates are not used - Removed "--altlookaside" option, as the above features make it redundant - Automatic tagless: If a proper git version/imports tag isn't found, a "tagless-mode" import will be attempted automatically at run-time - taglessmode command line option kept in case user still wants to force a tagless import -Skip Grube	2023-02-22 23:23:47 -05:00
Mustafa Gezen	06ffd1e507	Support ~ and % in Release	2022-11-15 15:14:59 +01:00
Mustafa Gezen	29778c7ea2	Proper NVR matching	2022-11-06 04:54:35 +01:00
Mustafa Gezen	2a2b10ba10	Formatting	2022-11-06 04:53:02 +01:00
Mustafa Gezen	e859f54e5e	Supported Go version is now 1.18	2022-11-03 04:39:57 +01:00
Mustafa Gezen	f3ac46447d	Stop converting v3 to v2	2022-11-03 04:36:25 +01:00
skip77	cff0cc0fbf	feature: "tagless import" feature mode (#4 ) In order to build from lookahead sources, srpmproc must support the new gitlab address for centos 9 stream as well as manipulating the sources to expected format. Resolves #2. - Options for tagless import and new stream lookaside format added - Git Mode now supports scanning for branch head in addition to the "normal" pulling of specific version tags - Alternate ProcessRPM added for Tagless mode in process.go (still a WIP) - Tagless mode converts a repo to the "traditional" format (SPECS/ + SOURCES/ + <pkg>.metadata ) - Tagless mode will build a dummy srpm to determine NVR information (rpmbuild + rpm shell commands) (will use this to tag imports in the target git) - Limitation: Tagless imports only pull the latest head from a branch - CentOS-Stream import branches are converted from stream-<MODULE_VERSION>-<RHEL_VERSION> to the more familiar r9s-stream-<VERSION> - stream-style YAML is detected and converted for modules, similar to the older modulemd.src.txt files - This new pattern is for "tagless mode" only, previous tagged imports (from git.centos.org) should not be affected	2022-09-27 23:32:50 -04:00
Mustafa Gezen	9b44bc6e50	Merge pull request #5 from jarod-w/main	2022-09-12 21:13:13 +02:00
jarod.w	d7ed1c7a7f	Fix the issue that nothing is ever written to tmpfs location when the tmpfs is absolute path. The test case of absolute path is below: [root@rockylinux86 ~]# srpmproc --version 8 --upstream-prefix file:///root/rocky --storage-addr file:///tmp/srpmproc-cache --source-rpm sed --tmpfs-mode /root/rocky/rpms/sed [root@rockylinux86 ~]# ls /root/rocky/rpms/sed/ r8 r8-beta r8s The test case of relative path is below: [root@rockylinux86 ~]# pwd /root [root@rockylinux86 ~]# srpmproc --version 8 --upstream-prefix file:///root/rocky --storage-addr file:///tmp/srpmproc-cache --source-rpm sed --tmpfs-mode ./rocky/rpms/sed [root@rockylinux86 ~]# ls /root/rocky/rpms/sed/ r8 r8-beta r8s Signed-off-by: jarod.w <wl.jarod@gmail.com>	2022-09-11 17:52:36 +08:00
Neil Hanlon	1c2091a489	Merge pull request #3 from NeilHanlon/main Add s390x and ppc64le to builds	2022-06-03 13:57:11 -04:00
Neil Hanlon	aacedb189f	Add s390x and ppc64le to builds	2022-06-03 13:56:05 -04:00
Mustafa Gezen	6df2fe1e3f	Fix NVR regex	2022-05-10 16:32:31 +02:00
Mustafa Gezen	e889daa21f	Regex arguments should be escaped	2022-05-10 12:02:37 +02:00
Mustafa Gezen	3f4c4ad211	Fetch should conform to metadata format	2022-05-10 10:34:02 +02:00
Mustafa Gezen	6ab7f9f00a	Metadata file not being found should throw an error	2022-05-10 09:35:48 +02:00
Mustafa Gezen	16ad80fdeb	Support inconsistent metadata naming and case-insensitive tags	2022-05-09 19:57:43 +02:00
Mustafa Gezen	8199a79889	Add support for version/release based fetching	2022-05-05 04:25:07 +02:00
Mustafa Gezen	f7017a9eac	Fix NVR regex for versions with a percentage, and fix dist regex for any EL	2022-05-02 10:39:42 +02:00
Mustafa Gezen	dd7fd31ebb	retry fetching patch repo if the error is invalid auth method	2022-04-24 15:53:36 +02:00
Mustafa Gezen	bc9d81bd8e	Strict branch mode should still include module streams	2022-04-23 22:56:39 +02:00
Mustafa Gezen	a84d5e5b62	fix non strict branch mode	2022-04-22 01:44:18 +02:00
Mustafa Gezen	45ad998d41	non-tmpfs mode should use memfs	2022-04-21 07:39:10 +02:00
Mustafa Gezen	a97ccb698b	add support for modulemd v3	2022-04-21 07:25:44 +02:00
Mustafa Gezen	61eaa8b8f8	fix tmpfs mode, replace stream mode with branch suffix and strict mode	2022-04-21 06:30:33 +02:00
Mustafa Gezen	8f55eb397f	enhancement: add support for downloading from blob storage instead of http endpoint	2022-04-03 04:29:49 +02:00