============= github-backup

|PyPI| |Python Versions|

The package can be used to backup an entire Github <https://github.com/>_ organization, repository or user account, including starred repos, issues, discussions and wikis in the most appropriate format (clones for wikis, json files for issues and discussions).

Requirements

• Python 3.10 or higher
• GIT 1.9+

Installation

Using PIP via PyPI::

pip install github-backup

Using PIP via Github (more likely the latest version)::

pip install git+https://github.com/josegonzalez/python-github-backup.git#egg=github-backup

Install note for python newcomers:

Python scripts are unlikely to be included in your $PATH by default, this means it cannot be run directly in terminal with $ github-backup ..., you can either add python's install path to your environments $PATH or call the script directly e.g. using $ ~/.local/bin/github-backup.*

Basic Help

Show the CLI help output::

github-backup -h

CLI Help output::

github-backup [-h] [-t TOKEN_CLASSIC] [-f TOKEN_FINE] [--token-from-gh]
              [-q] [--as-app] [-o OUTPUT_DIRECTORY] [-l LOG_LEVEL] [-i]
              [--incremental-by-files]
              [--starred] [--all-starred] [--starred-skip-size-over MB]
              [--watched] [--followers] [--following] [--all]
              [--issues] [--issue-comments] [--issue-events] [--pulls]
              [--pull-comments] [--pull-reviews] [--pull-commits]
              [--pull-details]
              [--labels] [--hooks] [--milestones] [--security-advisories]
              [--discussions] [--repositories] [--bare] [--no-prune]
              [--lfs] [--wikis] [--gists] [--starred-gists]
              [--skip-archived] [--skip-existing]
              [-L [LANGUAGES ...]] [-N NAME_REGEX] [-H GITHUB_HOST]
              [-O] [-R REPOSITORY] [-P] [-F] [--prefer-ssh] [-v]
              [--keychain-name OSX_KEYCHAIN_ITEM_NAME]
              [--keychain-account OSX_KEYCHAIN_ITEM_ACCOUNT]
              [--releases] [--latest-releases NUMBER_OF_LATEST_RELEASES]
              [--skip-prerelease] [--assets]
              [--skip-assets-on [SKIP_ASSETS_ON ...]] [--attachments]
              [--throttle-limit THROTTLE_LIMIT]
              [--throttle-pause THROTTLE_PAUSE]
              [--exclude [EXCLUDE ...]] [--retries MAX_RETRIES]
              USER

Backup a github account

positional arguments:
  USER                  github username

options:
  -h, --help            show this help message and exit
  -t, --token TOKEN_CLASSIC
                        personal access, OAuth, or JSON Web token, or path to
                        token (file://...)
  -f, --token-fine TOKEN_FINE
                        fine-grained personal access token (github_pat_....),
                        or path to token (file://...)
  --token-from-gh       read token from GitHub CLI (gh auth token)
  -q, --quiet           supress log messages less severe than warning, e.g.
                        info
  --as-app              authenticate as github app instead of as a user.
  -o, --output-directory OUTPUT_DIRECTORY
                        directory at which to backup the repositories
  -l, --log-level LOG_LEVEL
                        log level to use (default: info, possible levels:
                        debug, info, warning, error, critical)
  -i, --incremental     incremental backup
  --incremental-by-files
                        incremental backup based on modification date of files
  --starred             include JSON output of starred repositories in backup
  --all-starred         include starred repositories in backup [*]
  --starred-skip-size-over MB
                        skip starred repositories larger than this size in MB
  --watched             include JSON output of watched repositories in backup
  --followers           include JSON output of followers in backup
  --following           include JSON output of following users in backup
  --all                 include everything in backup (not including [*])
  --issues              include issues in backup
  --issue-comments      include issue comments in backup
  --issue-events        include issue events in backup
  --pulls               include pull requests in backup
  --pull-comments       include pull request review comments in backup
  --pull-reviews        include pull request reviews in backup
  --pull-commits        include pull request commits in backup
  --pull-details        include more pull request details in backup [*]
  --labels              include labels in backup
  --hooks               include hooks in backup (works only when
                        authenticated)
  --milestones          include milestones in backup
  --security-advisories
                        include security advisories in backup
  --discussions         include discussions in backup
  --repositories        include repository clone in backup
  --bare                clone bare repositories
  --no-prune            disable prune option for git fetch
  --lfs                 clone LFS repositories (requires Git LFS to be
                        installed, https://git-lfs.github.com) [*]
  --wikis               include wiki clone in backup
  --gists               include gists in backup [*]
  --starred-gists       include starred gists in backup [*]
  --skip-archived       skip project if it is archived
  --skip-existing       skip project if a backup directory exists
  -L, --languages [LANGUAGES ...]
                        only allow these languages
  -N, --name-regex NAME_REGEX
                        python regex to match names against
  -H, --github-host GITHUB_HOST
                        GitHub Enterprise hostname
  -O, --organization    whether or not this is an organization user
  -R, --repository REPOSITORY
                        name of repository to limit backup to
  -P, --private         include private repositories [*]
  -F, --fork            include forked repositories [*]
  --prefer-ssh          Clone repositories using SSH instead of HTTPS
  -v, --version         show program's version number and exit
  --keychain-name OSX_KEYCHAIN_ITEM_NAME
                        OSX ONLY: name field of password item in OSX keychain
                        that holds the personal access or OAuth token
  --keychain-account OSX_KEYCHAIN_ITEM_ACCOUNT
                        OSX ONLY: account field of password item in OSX
                        keychain that holds the personal access or OAuth token
  --releases            include release information, not including assets or
                        binaries
  --latest-releases NUMBER_OF_LATEST_RELEASES
                        include certain number of the latest releases; only
                        applies if including releases
  --skip-prerelease     skip prerelease and draft versions; only applies if
                        including releases
  --assets              include assets alongside release information; only
                        applies if including releases
  --skip-assets-on [SKIP_ASSETS_ON ...]
                        skip asset downloads for these repositories
  --attachments         download user-attachments from issues, pull requests,
                        and discussions
  --throttle-limit THROTTLE_LIMIT
                        start throttling of GitHub API requests after this
                        amount of API requests remain
  --throttle-pause THROTTLE_PAUSE
                        wait this amount of seconds when API request
                        throttling is active (default: 30.0, requires
                        --throttle-limit to be set)
  --exclude [EXCLUDE ...]
                        names of repositories to exclude
  --retries MAX_RETRIES
                        maximum number of retries for API calls (default: 5)

Usage Details

Authentication

GitHub requires token-based authentication for API access. Password authentication was removed in November 2020 <https://developer.github.com/changes/2020-02-14-deprecating-password-auth/>_.

The positional argument USER specifies the user or organization account you wish to back up.

Fine-grained tokens (-f TOKEN_FINE) are recommended for most use cases, especially long-running backups (e.g. cron jobs), as they provide precise permission control.

Classic tokens (-t TOKEN) are slightly less secure <https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#personal-access-tokens-classic>_ as they provide very coarse-grained permissions.

If you already authenticate with the GitHub CLI <https://cli.github.com/>_, you can use --token-from-gh to read the token with gh auth token instead of passing a token directly. This avoids placing the token in shell history or process arguments. When --github-host is set, the token is read with gh auth token --hostname HOST.

Fine Tokens

You can "generate new token", choosing the repository scope by selecting specific repos or all repos. On Github this is under *Settings -> Developer Settings -> Personal access tokens -> Fine-grained Tokens*

Customise the permissions for your use case, but for a personal account full backup you'll need to enable the following permissions:

**User permissions**: Read access to followers, starring, and watching.

**Repository permissions**: Read access to contents, discussions, issues, metadata, pull requests, and webhooks.


GitHub Apps

GitHub Apps are ideal for organization backups in CI/CD. Tokens are scoped to specific repositories and expire after 1 hour.

One-time setup:

1. Create a GitHub App at Settings -> Developer Settings -> GitHub Apps -> New GitHub App
2. Set a name and homepage URL (can be any URL)
3. Uncheck "Webhook > Active" (not needed for backups)
4. Set permissions (same as fine-grained tokens above)
5. Click "Create GitHub App", then note the App ID shown on the next page
6. Under "Private keys", click "Generate a private key" and save the downloaded file
7. Go to Install App in your app's settings
8. Select the account/organization and which repositories to back up

CI/CD usage with GitHub Actions:

Store the App ID as a repository variable and the private key contents as a secret, then use actions/create-github-app-token::

- uses: actions/create-github-app-token@v1
  id: app-token
  with:
    app-id: ${{ vars.APP_ID }}
    private-key: ${{ secrets.APP_PRIVATE_KEY }}

- run: github-backup myorg -t ${{ steps.app-token.outputs.token }} --as-app -o ./backup --all

Note: Installation tokens expire after 1 hour. For long-running backups, use a fine-grained personal access token instead.

Prefer SSH

If cloning repos is enabled with ``--repositories``, ``--all-starred``, ``--wikis``, ``--gists``, ``--starred-gists`` using the ``--prefer-ssh`` argument will use ssh for cloning the git repos, but all other connections will still use their own protocol, e.g. API requests for issues uses HTTPS.

To clone with SSH, you'll need SSH authentication setup `as usual with Github <https://docs.github.com/en/authentication/connecting-to-github-with-ssh>`_, e.g. via SSH public and private keys.


Using the Keychain on Mac OSX

Note: On Mac OSX the token can be stored securely in the user's keychain. To do this: