thomaskekeisen.de

From the life of a screen worker

Attention: This page contains partner and advertising links. Therefore this page is to be understood entirety as an advertisement!

Synchronize with external repository

For a project in my company I had to synchronize an own Bitbucket repository with an external one. My idea was to accomplish this using Bitbucket Pipelines. Here is an explanation how its possible to synchronize two Bitbucket repositories using Bitbucket Pipelines.

1: Create access keys for both repositories

To access your Bitbucket repository without an real user account you can add an access key directly in your repository settings on Bitbucket. Basically the access key is a public key created using the command ssh-keygen . If you want to directly pass a file name for the keys you can use the parameter -f , e.g. ssh-keygen -f example .

Screenshot: Generate a new ssh key
Screenshot Generate a new ssh key

You can just type cat example.pub to get the contents of your new generated access key.

Screenshot: Show the newly generated public key
Screenshot Show the newly generated public key

2: Add the access keys to Bitbucket

Once you have created your access keys, you have to add them into your Bitbucket repositories. You can use the same access key for both repositories but I generated two different keys and called them source and target . So just head to Settings > Access keys and add your keys.

Screenshot: Add your access keys to Bitbucket
Screenshot Add your access keys to Bitbucket

3: Configure your pipeline

My bitbucket-pipelines.yml contains this little script. I use php:7.1-fpm as image since my pipeline runs some more application-specific stuff for some php application deployment.

                        bitbucket-pipelines.yml
                        
                            image: php:7.1-fpm

                            pipelines:
                                default:
                                    -   step:
                                            script:
                                                - apt-get update && apt-get install -y unzip git

                                                - sh .pipelines/synchronize-repositories.sh

                                                - echo "Repository has been successfully synchronized"

The file .pipelines/synchronize-repositories.sh then looks like the following:

                        .pipelines/synchronize-repositories.sh
                        
                            #!/bin/sh

                            SOURCE_REPOSITORY_NAME=source-repository.git
                            SOURCE_URL=git@bitbucket.org:team1/$SOURCE_REPOSITORY_NAME
                            TARGET_REPOSITORY_NAME=target-repository.git
                            TARGET_URL=git@bitbucket.org:team2/$TARGET_REPOSITORY_NAME

                            echo "Current path is"
                            pwd

                            eval "$(ssh-agent)"

                            echo "Enabling source ssh key"
                            chmod 600 .ssh/source
                            # Only for local debugging
                            # git config --local core.sshCommand "/usr/bin/ssh -i .ssh/source"
                            ssh-add .ssh/source

                            if [ "$?" != "0" ]; then
                              echo "Failed to load source ssh key"
                              exit 1
                            fi

                            echo "Deleting any old source that may exist"
                            rm -rf $SOURCE_REPOSITORY_NAME

                            echo "Checking out the source repository"
                            git clone --bare $SOURCE_URL

                            if [ "$?" != "0" ]; then
                              echo "Failed to clone source"
                              exit 1
                            fi

                            echo "Entering the checked out repository"
                            cd $SOURCE_REPOSITORY_NAME

                            echo "Downloading the source repository"
                            git fetch origin --tags

                            if [ "$?" != "0" ]; then
                              echo "Failed to fetch origin"
                              exit 1
                            fi

                            echo "Enabling the target ssh key"
                            chmod 600 ../.ssh/target
                            # Only for local debugging
                            # git config --local core.sshCommand "/usr/bin/ssh -i ../.ssh/target"
                            ssh-add ../.ssh/target

                            if [ "$?" != "0" ]; then
                              echo "Failed to load target ssh key"
                              exit 1
                            fi

                            echo "Adding the target as mirror remote"
                            git remote add --mirror=fetch target $TARGET_URL

                            if [ "$?" != "0" ]; then
                              echo "Failed to add target as remote: $TARGET_URL"
                              exit 1
                            fi

                            if [ "$?" != "0" ]; then
                              echo "Failed to fetch target"
                              exit 1
                            fi

                            echo "Copying all data from the source to the target repository"
                            git push target --all
                            git push target --tags

                            if [ "$?" != "0" ]; then
                              echo "Failed to push to target"
                              exit 1
                            fi

                            cd ..

                            echo "Deleting any old source that may exist"
                            rm -rf $SOURCE_REPOSITORY_NAME

                            # Only for local debugging
                            # git config --local core.sshCommand "/usr/bin/ssh"

At the end this script just does a one-way synchronization from SOURCE_URL/SOURCE_REPOSITORY_NAME to TARGET_URL/TARGET_REPOSITORY_NAME . Make sure you add the correct values matching your environment and place your ssh keys generated in step 1 in .ssh/source and .ssh/target . Of course you can modify this script to use environment-variables but this was not necessary in my case since the access keys are only used for the synchronization of the same application code.

So with the pipeline configuration shown above, the repository in TARGET_URL gets completely synchronized with the repository in SOURCE_URL whenever a branch was pushend in SOURCE_URL . In other words: The copied code in the backup repository of our customer is up to date in realtime.

Screenshot: The pipeline is running successfully
Screenshot The pipeline is running successfully

Links

Share

Comments

This is the printed version of: https://thomaskekeisen.de/en/blog/git-bitbucket-pipelines-synchronize-backup