Streamlining Development through Monorepo with Independent Release Cycles

Pramita Winata

Streamlining Development through Monorepo with Independent Release Cycles

In the ever-evolving world of software development, the choice of repository architecture plays a crucial role in the efficiency and scalability of the project. Similar to numerous challenges in development, there isn’t a pre-established solution for determining the most suitable approach. The choice between strategies depends on the specific circumstances of each company and project.

In our recent engagement, we need to build and manage computation models stored as Docker images, which we commonly referred to as computation-image-models. Each computation-image-model runs as an independent application. Thus it has its own release cycle and dependencies. Our main focus is to have a simplistic and easily maintainable repository architecture due to the lack of resources.

In this article, we are going to go into the detail on how we choose our strategy and explore the benefits and best practices associated with this approach.

Understanding Monorepos

In the multirepo approach, projects are typically organized into separate repositories based on functionalities or modules. However, this approach invites challenges such as versioning inconsistencies, complex dependency management, inconsistencies in the deployment process and difficulties in sharing code across projects.

Monorepository, or monorepo for short, is a single repository containing multiple distinct projects, with well-defined relationships. This approach offers several benefits, such as simplified dependency management, code sharing, centralized configuration and easier collaboration. However, releasing multiple artifacts from a monorepo can be a challenging task.

Implementing Independent Release Cycles

An interesting aspect of utilizing a monorepo is the ability to bring together various projects in a single location. Each project can operate independently yet collaboratively, while maintaining a central configuration. The challenge lies in managing the release process when dealing with multiple projects or artifacts within the same codebase. A well-defined multiple artifacts releasing strategy is essential to leverage the benefits of monorepos without compromising on release efficiency.

Here are some strategies we implemented for independent release cycles in the monorepo:

• Versioning and Tagging

Implement a versioning strategy that allows each project to have its own version number. This ensures that changes in one project don’t accidentally affect others during the release process. By utilizing tags or labels within the monorepo, teams can track and manage the release status of individual projects without affecting others.

• Automated CI/CD Pipelines

Implement automated continuous integration and continuous delivery (CI/CD) pipelines that are tailored to each project’s needs. This ensures that changes to a specific project trigger only the relevant build and deployment processes, minimizing the impact on other projects.

• Dependency Management

Utilize dependency management tools to handle the dependencies between projects efficiently. This includes specifying version ranges and ensuring that each project can consume the necessary versions of shared libraries or components.

• Isolation through workspace configuration

Leverage workspace configurations, i.e., virtual environment, to isolate the development environment for each project within the monorepo. This allows developers to work on individual projects without interfering with others, providing a clear boundary for dependencies and builds.

• Documentation and Communication

Document the release process for each project and establish clear communication channels. This ensures that all team members are aware of the release schedules, versioning policies, and any specific instructions related to individual projects.

We provided a sample repository as development reference based on our project experience.

Release tool

There are various tools option available to help with automating this release process. We opted to use manifest driven release-please. This tool automates CHANGELOG generation, the creation of GitHub releases, and version bumps for your projects.

Dependency management tool

Depending on the programming language, there are diverse options for dependency management tools. In our scenario, we opt for poetry as our preferred tool. With poetry, dependencies are specified in the pyproject.toml file and subsequently locked in the poetry.lock file. In the sample repository, the pyproject.toml file serves as a means to express dependencies between Project A and Project B, specifying their compatible versions. A snippet of the pyproject.toml can be observed below.

[tool.poetry.dependencies]
python = ">=3.9.10, <3.11.0"

[tool.poetry.group.dev.dependencies]
black = "^23.3.0"
pytest = "^7.3.1"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

[[tool.poetry.source]]
name = "PyPI"
priority = "primary"

Repository structure

We organized each project into a separate folder, complete with its own CI/CD configuration. This approach enables us to have precise control over the build, testing, and deployment processes for each individual project. Moreover, it facilitates smoother team collaboration, as each project is encapsulated within its designated space.

.
├───.github
│   └───workflows
│   │   ├───pr_a.yml
│   │   ├───deploy_a.yml
│   │   ├───pr_b.yml
│   │   └───deploy_b.yml
└───model
    ├───a
    │   ├───src
    │   ├───tests
    │   ├───README.md
    │   └───CHANGELOG
    └───b
    │   ├───src
    │   ├───tests
    │   ├───README.md
    │   └───CHANGELOG

CI/CD

The depicted diagram outlines the CI/CD process. It initiates upon the merging of a pull request, and subsequently, the release-please tool assesses whether a new version release is necessary. If deemed necessary, an automatic creation of the release-PR follows suit.

Sample of a release-PR is shown below.

Once the release-PR is approved GitHub tag and GitHub release will be created.

The manifest is a crucial part of this workflow as it determines the projects that needs to be tracked in the repository. The manifest is configured as a json file as shown below.

{
  "packages": {
    "model/a": {
      "changelog-path": "CHANGELOG.md",
      "release-type": "python",
      "component": "a"
    },
    "model/b": {
      "changelog-path": "CHANGELOG.md",
      "release-type": "python",
      "component": "b"
    }
  }
}

An illustration of the version bumping determination is provided in the code snippet below. Each project will have its own output that can be used for subsequent steps. The complete code is available in release.yml.

  - name: Release with release-please
    uses: google-github-actions/release-please-action@v3
    id: release
    with:
      command: manifest
      monorepo-tags: true
      pull-request-header: ":robot: I have created a release"
  - name: "Generate release-please output"
    id: create_output
    shell: bash
    run: |
      echo "a-release_created=$(echo ${{ steps.release.outputs['model/a--release_created'] }})" >> "$GITHUB_OUTPUT"
      echo "a-tag_name=$(echo ${{ steps.release.outputs['model/a--tag_name'] }})" >> "$GITHUB_OUTPUT"
      echo "b-release_created=$(echo ${{ steps.release.outputs['model/b--release_created'] }})" >> "$GITHUB_OUTPUT"
      echo "b-tag_name=$(echo ${{ steps.release.outputs['model/b--tag_name'] }})" >> "$GITHUB_OUTPUT"

Executing release-please triggers the automated creation of a GitHub tag, GitHub release, and changelog. This workflow is highly customizable and can be easily expanded. As an illustration in the sample reference, we enhanced the release process by the publication of an image to the Azure Container Registry(ACR) whenever a specific project undergoes a new release. As an example, we use the previous step output of needs.release_job.outputs.a-release_created and needs.release_job.outputs.a-tag_name to published the correct image to ACR.

  build_publish_a_image:
    name: Build Publish a Image
    uses: ./.github/workflows/docker.yml
    needs: release_job
    if: needs.release_job.outputs.a-release_created == 'true'
    with:
      image_tag: ${{ needs.release_job.outputs.a-tag_name }}
      build_context: "a"

Conventional commit

Since the release-please tools relies heavily on Conventional commits, we lint our commit messages using commitlint. Full code can be found in style.yml

jobs:
  CheckCommitStyle:
    runs-on: ubuntu-latest
    if: ${{ github.event_name == 'pull_request' }}
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Install commitlint dependencies
        run: |
            npm i --global "@commitlint/config-conventional@17" "@commitlint/cli@17"
        shell: bash
      - name: Conventional Commit Linting
        run: |
          # Linting commits from source branch tip to target branch tip
          commitlint --from ${{ github.event.pull_request.base.sha }} --to ${{ github.event.pull_request.head.sha }} --verbose --config ./.github/config/commitlint.config.js
        shell: bash

Conclusion

Just like with many challenges in development, there’s no one-size-fits-all solution for figuring out the best way to do things. It really depends on what works for each company and project. However, teaming up a monorepo with independent release cycles seems like a great solution for companies wanting to make their development processes simplified and smoother. When we bring all our projects into a single repository and let each one have its own release schedule, we get the perks of better teamwork, sharing code easily, and getting things to market faster. Pulling it off successfully means being all in on automation, keeping the communication channels open, and being smart about versioning and managing dependencies.

Pramita Winata

Follow