Effortless Git Submodule Sync with GitHub Actions

Starting point

I have a private repository src/content that I want to add as a submodule to another repository. The “other” repository is called the parent repository. Consider following structure of the parent repository:

├── src/
│   ├── components/
│   │   ├── <folder>
│   │   │   ├── <file>
│   │   │   └── <file>
│   │   ├── ...
│   │   ├── <folder>
│   ├── content/  👈 the private repo (reference) should be contained in this folder 
├── ...

Each time the private repository is updated, the parent repository should be updated as well. This will be achieved by using GitHub Actions.

👆 The challenges

1. How can I add a private repository as a submodule to the parent repository?
2. How can I automate the update process of the submodule (aka the private repository)?
3. How can I ensure that the parent repository is updated each time the private repository is updated?

How to solve the challenges

Requirements

You will need the following “ingredients” to solve the challenge:

• GitHub account
• GitHub repository
• GitHub Actions (to automate the update process of the submodule)

Instructions

① Configure Github account

First, prepare your Github account. You need to generate a personal access token (PAT) which will be used afterwords within connections in order to access the private repository from inside a workflow (Github action). To do so, follow these steps:

1. Login into your Github account and go to GitHub Settings > Developer Settings > Personal Access Tokens > Fine-grained tokens.
2. After clicking on ‘Generate new token’, create a new PAT:
   
   $\rightarrow$ Give a meaningful name (e.g. PRIVATE_REPO_SYNC) that corresponds to the secret’s definition in the previous step and select the desired repository (in this case the private repository that should be added as a submodule).
3. Under the ‘Permissions’ section, define the appropriate access permissions for Contents > Access: Read-only.
4. Let Github generate the token and copy it for further use in the next action.
5. Go to the parent’s repository settings > Actions secrets and variables > Repository secrets. Add a new repository secret with the same name as the PAT (e.g. PRIVATE_REPO_SYNC) and paste the PAT value.
   $\rightarrow$ Make sure to choose ‘Repository secrets’ and not ‘Environment secrets’ as the latter are not accessible in the workflow.

You will need the gnerated PAT in the next steps within the workflow (step ②) and the Git submodules definition (step ③).

② Configure the workflow as GitHub Action

Define your deployment workflow according to you needs – as a easy-to-read yaml file. 💪

It is important to add an action step that references the name of the personal access token (PAT). You will need in the token field to access the private repository via Github action by using the connection that you will set up in the next step. Any lack of definition will lead to an error when trying to establish the connection to the private repository during the workflow run, as the commonly (“internally”) used GITHUB_TOKEN does not have access to private repositories.

- uses: actions/checkout@v4
  with:
    submodules: 'recursive'
    token: ${{ secrets.PRIVATE_REPO_SYNC }}

Implement another action step to initialize and update the submodules (which we are going to add in the next step) in the parent repository. This step will ensure that the parent repository is always up-to-date with the private repository:

- name: Initialize and update submodules
  run: |
    git submodule init
    git submodule update --remote --recursive

To put it all together, here is the discussed part of a workflow file, which you can extend according to your needs:

    # ... further actions omitted for brevity

    - uses: actions/checkout@v4
      with:
        submodules: 'recursive'
        token: ${{ secrets.PRIVATE_REPO_SYNC }}
    
    # ... further actions omitted for brevity
    
    - name: Initialize and update submodules
      run: |
        git submodule init
        git submodule update --remote --recursive
    
    # ... further actions omitted for brevity

③ Configure project folder as Git submodule

To conclude, include the private repository as a submodule that will reside in your code base. In our example, it should be visible in the src/content folder within the parent’s repository structure. To do so, execute the following command in the parent repository to add the private repository (using a HTTPS connection) as a submodule:

git submodule add https://USERNAME:TOKEN@github.com/<owner>/<repo>.git src/content

Exchange <owner> and <repo> with the owner’s name (in my case tmaestrini) and the name of the private repository that you want to include as submodule.

The above action will add following lines to the .gitmodules file:

[submodule "src/content"]
    path = src/content
    url = https://USERNAME:TOKEN@github.com/owner/repo.git

Wrapping up

Including a private repository as a submodule in a parent repository is a very useful use case. By using this approach, you can ensure that the parent repository always contains the latest information / data from the private repository. You have to make sure that the private repository is accessible via a personal access token (PAT) and that the PAT is stored as a secret in the parent repository. GitHub Actions can then be used to automate the whole update process.

Remember the following key points:

• Use GitHub Actions to automate the whole update process.
• You can use HTTPS or SSH for connection to submodules with private repositories.
• GITHUB_TOKEN doesn’t have access to private repos; therefore, you need to use a personal access token (PAT) to access the private repository.
• Never commit token directly in .gitmodules, use GitHub Secrets instead.