First version of the Github2Synology script

LICENSE.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2017 Bairwell Ltd/Richard Bairwell.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,52 @@
+Github2Synology
+===============
+A simple Ash (practically "BusyBox bash") script designed to run on the Synology DS range of file storage servers to backup all repositories (and wikis) for a user from Github.
+
+Running
+=======
+* Ensure you have `git` installed on the Synology - this can be download from the SynoCommunity. The script also needs cUrl and [jq](https://stedolan.github.io/jq/) but these seem standard on Synologys.
+
+* Now login to Github and go to [https://github.com/settings/tokens]( https://github.com/settings/tokens) and create a personal access token with the following scopes:
+    - repo (repo itself including all subs) - Full control of private repositories
+    - admin:org read:org - Read org and team membership
+
+* Add this token as the OAUTH_TOKEN in line 7 on the `github2synology.sh` script. (`OAUTH_TOKEN="[PUT YOUR TOKEN HERE BETWEEN THE QUOTES]"`)
+* Ensure the backup path is correct on line 9. (`BACKUP_PATH="/volume1/serverBackups/github/backup"`)
+* Copy the script over to your Synology and run it (all via SSH)
+
+Problems?
+=========
+Getting "Access forbidden/Repository not found" issues?
+-------------------------------------------------------
+This is because the Synology doesn't have access to your Github repositories. The "best (most secure)" way to resolve this is just to enable "SSH Key forwarding" from your Mac/PC to the Synology so it uses your SSH keys for authentication.
+
+On Mac and Linux command line, you should be able to just create/edit `~/.ssh/config` and add:
+
+```
+Host [synology]
+   ForwardAgent yes
+```
+   
+(replacing synology with the IP/name of your Synology)
+
+In PuTTy on Windows, this is under `Connection`->`SSH`->`Auth`->`Authentication parameters : Allow Agent Forwarding` (ensure Pagaent is running with a Github recognised key).
+
+To test if this is setup correctly, try running from the Synology:
+```ssh -T git@github.com```
+you should get back:
+```Hi xxxxx! You've successfully authenticated, but GitHub does not provide shell access.```
+
+"It's only backing up 100 repositories, I've access to more"
+------------------------------------------------------------
+Due to the script's simplicity, it does NOT currently read the Github provided `Link:` Http headers which give the `next` page details.
+
+To work around this, run the command `curl -I "https://api.github.com/user/repos?type=all&page=1&per_page=100" -H "Authorization: token [OAUTHTOKEN]"` (replacing `[OAUTHTOKEN]` with your token). You'll then see a line such as:
+`Link: <https://api.github.com/user/repos?type=all&per_page=100&page=2>; rel="next", <https://api.github.com/user/repos?type=all&per_page=100&page=3>; rel="last"`
+
+Then just add the Link:...rel="next" entry to the bottom of the script such as:
+```
+API_URL="https://api.github.com/user/repos?type=all&per_page=100&page=2"
+
+fetch_fromUrl
+```
+and the `last` and `next` do not match, just repeat these steps changing the `&page=1` increment in the curl command.

github2synology.sh

@@ -0,0 +1,61 @@
+#!/bin/sh
+# A script to backup Github repositories to a Synology.
+# By Richard Bairwell. http://www.bairwell.com
+# MIT Licenced. https://github.com/bairwell/github2synology
+
+# token from https://github.com/settings/tokens
+OAUTH_TOKEN="[PUT YOUR TOKEN HERE BETWEEN THE QUOTES]"
+# where should the files be saved
+BACKUP_PATH="/volume1/serverBackups/github/backup"
+
+
+# you shouldn't need to change anything below here - unless you have over 100 repos: in which case, see the bottom.
+
+API_URL="https://api.github.com/user/repos?type=all&per_page=100"
+GIT="/volume1/@appstore/git/bin/git"
+
+fetch_fromUrl() {
+    echo "Fetching from ${API_URL}"
+    REPOS=`curl -H "Authorization: token ${OAUTH_TOKEN}" -s "${API_URL}" | jq -r 'values[] | "\(.full_name),\(.private),\(.git_url),\(.has_wiki)"'`
+    for REPO in $REPOS
+    do
+        REPONAME=`echo ${REPO} | cut -d ',' -f1`
+        PRIVATEFLAG=`echo ${REPO} | cut -d ',' -f2`
+        ORIGINALGITURL=`echo ${REPO} | cut -d ',' -f3`
+        HASWIKI=`echo ${REPO} | cut -d ',' -f4`
+        GITURL="${ORIGINALGITURL/git:\/\/github.com\//git@github.com:}"
+        mkdir "${BACKUP_PATH}/${REPONAME}" -p
+        REPOPATH="${BACKUP_PATH}/${REPONAME}/code"
+
+
+        if [ -d "$REPOPATH" ]; then
+            echo "PULLING Repo URL: ${REPONAME} from url ${GITURL} to ${REPOPATH}"
+            cd ${REPOPATH}
+            ${GIT} pull
+        else
+            echo "CLONING Repo URL: ${REPONAME} from url ${GITURL} to ${REPOPATH}"
+            ${GIT} clone ${GITURL} ${REPOPATH}
+            if [ "true"===${PRIVATEFLAG} ]; then
+                `touch ${BACKUP_PATH}/${REPONAME}/private`
+            fi
+        fi
+        if [ "true"===${HASWIKI} ]; then
+            WIKIPATH="${BACKUP_PATH}/${REPONAME}/wiki"
+            WIKIURL="${ORIGINALGITURL/git:\/\/github.com\//git@github.com:}"
+            WIKIURL=`echo ${WIKIURL} | sed -e "s/.git$/.wiki.git/"`
+            if [ -d "$WIKIPATH" ]; then
+                echo "PULLING Repo Wiki: ${REPONAME} from url ${WIKIURL}: to ${WIKIPATH}"
+                cd ${WIKIPATH}
+                ${GIT} pull
+            else
+                echo "CLONING Repo Wiki: ${REPONAME} from url ${WIKIURL}:to ${WIKIPATH}"
+                ${GIT} clone ${WIKIURL} ${WIKIPATH}
+            fi
+        fi
+    done
+
+}
+fetch_fromUrl
+# If you need additional urls adding, put them here.
+#API_URL="https://api.github.com/user/repos?type=all&per_page=100&page=2"
+#fetch_fromUrl

test.sh

@@ -0,0 +1,6 @@
+#!/bin/sh
+OAUTH_TOKEN="bacd4353ca9a4f771f16dc7d0c16802de87ca025"
+API_URL="https://api.github.com/user/repos?type=all&per_page=100"
+
+HEADERS=`curl -sI -H "Authorization: token ${OAUTH_TOKEN}" -s "${API_URL}" | grep `
+NEXT=$