Advanced Workflows in GitHub Actions

Introduction

GitHub Actions is a managed platform to build your continuous integration and continuous delivery or deployment pipelines. It allows you to automate any build, test, deployment, and more jobs depending on your solution requirements. Integrating the workflows into the GitHub repositories will enable you to trigger them by commits, pull requests, or even manually through GitHub API and CLI.

The workflows can get complicated when you have more services and jobs to be done. A single workflow with simple jobs will not be enough in those cases. Therefore, it is good to improve your workflows with the advanced features. You can save time on implementation and execution with the good use of advanced workflows.

This article will share different advanced features provided by GitHub Actions. But before we start, there are some components you should know first.

Workflows

A workflow is where you configure the automated processes by defining jobs. Each workflow is a YAML file pushed to your GitHub repository in the `.github/workflows` directory. It will be executed when a predefined event is triggered manually or automatically. You can also schedule an execution.

Each GitHub repository has its workflow but is not limited to one. You can create multiple workflows depending on your solution. The workflows in a repository can also work with those in other repositories. For their details, we will explain them afterward.

Events

An event is an activity that triggers the execution of a workflow. You can use native GitHub triggers like pushing a commit, creating a pull request, and opening an issue. They also support the use of GitHub API or CLI, so that you can create the event from outside GitHub.

Jobs

A job is a set of steps or actions you want to execute. Those steps can be executable shell scripts or run-able actions. They are executed in order and dependent on each order. As each job is running in a runner, steps can share outputs and artifacts as input for the next steps.

In a workflow, it usually includes multiple jobs. They are running in parallel by default unless a job has dependency from other jobs. If this is the case, that job needs to wait until the dependent jobs have been completed.

Runners

A runner is a server that runs your job. There are two types, GitHub-managed and self-managed. GitHub-managed runners mean they are managed by GitHub, providing Microsoft Windows, Ubuntu Linux, and macOS runners. You can also set up self-managed runners if your jobs need to be executed privately. It offers flexibility if there is any compliance requirement in your organization.

Reusable Workflows

One of the main tasks in DevOps is to maximize the efficiency. In GitHub Actions, we should design reusable workflows as much as possible to avoid duplication. There are always repeating jobs in an automated pipeline. Instead of copying and pasting from one workflow to another, reusing workflows is the better option. It is also the best practice to build an organization-wise reusable workflow library.

Before showing the example, we need to learn some GitHub terms first. In GitHub language, a workflow uses another workflow is referred to as a `Caller` workflow. The workflow being used is referred to as a `Called` workflow, and this `Called` workflow is the one being reused.

We have prepared two `Caller` workflows and one `Called` workflow. It means that we reuse a workflow in two different workflows.

`Called` workflow:

name: Called Workflow
run-name: Called by ${{ inputs.environment_name }}
on:  
  workflow_call:    
    inputs:      
      environment_name:        
        description: 'The environment name'        
        required: true        
        type: string      
      message:        
        description: 'The message to display'        
        required: true        
        type: string

jobs:  
  reusable_job:    
    runs-on: ubuntu-latest    
    steps:      
      - name: Deploy to ${{ inputs.environment_name }}        
        run: echo '${{ inputs.message }}

QA `Caller` workflow:

name: QA Caller

on:
  workflow_dispatch:

jobs:
  call-reusable-workflow:
    uses: ./.github/workflows/called_workflow.yml
    with:
      environment_name: 'QA'
      message: 'I am for QA environment'

Production `Caller` workflow:

name: Production Caller

on:
  workflow_dispatch:

jobs:
  call-reusable-workflow:
    uses: ./.github/workflows/called_workflow.yml
    with:
      environment_name: 'Prod'
      message: 'I am for Production environment'

The above example is a typical usage of reusable workflow. The two `Caller` workflows provide a different set of variables as the input of the `Called` workflow. From the output screenshots, two different outputs show `QA` and `Production`. It is because of the variables in the `Called` which retrieve the values from the `Caller`. 

Job Dependencies

In most pipelines, jobs for different stages are usually executed in order, like building a Docker image before deployment and creating issues after testing. As mentioned, GitHub Actions executes jobs and workflows in parallel by default. For executing by order, we need to use the job dependency concept.

Using the `needs` field to identify any jobs that must be completed successfully before this job can start. The value in the field can be a string or an array of strings which means a job can depend on multiple jobs. If a job fails or is skipped for any reason, everything depending on this job will also be skipped unless there is a conditional expression allowing it to continue.

The following are some examples of job dependencies. 

Ex1 - If there are two jobs without defining prerequisite jobs, they will be running in parallel.

jobs:
  core-job:
    runs-on: ubuntu-latest
    steps:
      - run: echo 'I am the core job that always runs.'

  optional-job:
    runs-on: ubuntu-latest
    steps:
      - run: echo 'I only run if core-job has been completed successfully.'

Ex2 - If you define the prerequisite job, the jobs will be executed in order.

jobs:
  core-job:
    runs-on: ubuntu-latest
    steps:
      - run: echo 'I am the core job that always run.'

  optional-job:
    runs-on: ubuntu-latest
    needs: [core-job]
    steps:
      - run: echo 'I only run if core-job has been completed successfully.'

‍

Ex3 - If the prerequisite job has been failed, the next job will be skipped.

jobs:
  core-job:
    runs-on: ubuntu-latest
    steps:
      - run: exit 1

  optional-job:
    runs-on: ubuntu-latest
    needs: [core-job]
    steps:
      - run: echo 'I only run if core-job has been completed successfully.'

Environment Management

For most companies, there are multiple environments for different purposes. For example, a production environment for client-facing versions, a QA environment for testing, and a development environment for the implementation stage. As those environments are mutually exclusive, there are corresponding variables and secrets respectively.

To deal with it, we can use the Environment feature in GitHub Actions. You can configure variables, secrets, and protection rules for each environment. When the workflows of a specific environment have been triggered, it will check if prerequisite rules have been fulfilled, and then pull the necessary items for job execution. It can be used with reusable workflows that pass corresponding variables and secrets to the same set of workflows but execute on different environments.

jobs:
  qa:
    runs-on: ubuntu-latest
    environment: qa
    steps:
      - run: echo '${{ vars.MESSAGE }}'

  prod:
    runs-on: ubuntu-latest
    environment: prod
    steps:
      - run: echo '${{ vars.MESSAGE }}'

From the above example, two jobs executed the same command but had a different output, and the outputs are controlled at the environment level in GitHub Actions. The workflow can easily scale if there are more environments by changing the values in the `environment` field.

Advanced Triggering Options

GitHub is a platform with many powerful issues and project management features. They provide advanced triggering options in GitHub Actions, allowing you to create workflows that are responsive to code changes and can be scheduled, triggered by external events, and conditionally based on several criteria.

The following are some useful triggering options while designing and implementing your GitHub Actions workflows.

Scheduled Workflows. This kind of workflow is triggered at specific times using the cron syntax.

name: Scheduled Workflow

on:
  schedule:
    - cron: '0 0 * * *' # Runs at midnight every day

jobs:
  scheduled-job:
    runs-on: ubuntu-latest
    steps:
      - name: Run a scheduled task
        run: echo "This job runs every day at midnight"

Event-based Workflows. It is triggered by specific GitHub native events like issues, pull requests, and manually through UI or CLI.

name: Event-based Workflow

on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main
  issues:
    types: [opened, edited]
  workflow_dispatch:

jobs:
  event-job:
    runs-on: ubuntu-latest
    steps:
      - name: React to events
        run: echo "This job runs on push, pull request, and issue events"

External Events Workflows. Using custom events externally and trigger the workflows using GitHub API.

name: External Events Workflow

on:
  repository_dispatch:
    types: [external-event]

jobs:
  external-event-job:
    runs-on: ubuntu-latest
    steps:
      - name: Run on external event
        run: echo "This job runs on an external event"

The external events trigger a GitHub API like this,

curl -X POST -H "Accept: application/vnd.github.v3+json" -H "Authorization: token YOUR_PERSONAL_ACCESS_TOKEN" https://api.github.com/repos/OWNER/REPO/dispatches --data '{"event_type":" external-event"}'

The above options have provided tons of combinations on how your solution can be. You can also provide custom variables and secrets at the trigger level.

Conclusion

Using GitHub Actions in an advanced way allows DevOps Engineer to create a flexible and efficient CI/CD pipeline. Utilizing reusable workflows, job dependencies, environment management, and different triggering options can significantly enhance the automation processes.