PROJECT

Obsidian Knowledgebase with Git Sync

Project Overview

I’ve seen multiple tech personalities online say that Obsidian is the king of note-taking apps. I started out using ClickUp, then migrated to Git Book. Git Book left a little to be desired, and the prospect of having super fast access to my documents while having a ton of customisation was too much intrigue to pass up. The hurdle was always: that’s a lot of work to mitigate (I have a lot of documents), and how do I properly sync it? Obsidian have a paid sync service…. but I want to do it myself (for free).

I decided to take the plunge and make a proper project out of it. In this project, I will be detailing how I managed to get this all set up. Additionally, as I knew I wanted to do cross-device syncing with GitHub, I’ve upped the challenge by using the opportunity to also use Git in the cmd line while on my Work PC and laptop. Once this is all set up and functional, it opens the door to later using AI via an LLM to talk to your documents self-hosted for free. Let’s get a knowledgebase set up!

Obsidian

Firstly, on any device that my knowledgebase needs to be accessed on needs to have Obsidian installed (obviously).

Secondly, I set a single device (my home desktop) to be the primary device by which I can consider the master device that does the majority of the updates. The reason being to 1 – have some consistency, and 2 – to compensate for some difficult behaviour with the Obsidian Git Sync plugin across multiple devices (explained later).

Setting Up Obsidian

Firstly, create a new vault (where the files will be stored). Then, go to the cog in the bottom left which brings up the settings.

Appearance > Base Theme – set to dark mode, it’s the law.

Themes are presets to control how the pages look. I use the Git Hub theme.

Third Party Plugins > Enable these and click on Browse next to community plugins.

I’ve installed these three:

GitHub Sync <- for GitHub sync with a button press

Homepage <- creates a homepage you can use as a master contents page etc

Iconize <- to allow you to add emojis to page and folder names

You may have to enable them before they can be used.

This isn’t a complete list by any means. As I use this in the coming years, I’ll no doubt add more, but this is OK for me right now.

GitHub Repository

Next, I set up a private GitHub repository to host the documents. It would be a lot easier to make it public, but where is the fun in that, and I don’t want anyone seeing my documents.

When using the plugin in Obsidian, if you ever have issues, you can consult the Readme:

https://github.com/kevinmkchin/Obsidian-GitHub-Sync/blob/main/README.md

GitHub syncing via the git cmd line may prompt you to log in, but GitHub doesn’t actually allow you to simply log in to access a repo, so the GitHub process is two parts:

  • Creating the repository to host the documents
  • Creating SSH keys to allow for remote syncing

The sync method of the plugin seems to work with the https URL rather than the SSH URL. I’ve never got the SSH key URL working in the plugin for some reason.

Example https URL: https://github.com/thecyberdragon/obsidian-testing.git

GitHub Sync Plugin

In the plugin settings for GitHub sync, add the https URL into the first box.

The author states you have to do an initial “git init” before this can work, and he also recommends pushing to the remote repository (GitHub) to make sure you can.

Before setting up the repo, download and install Git the command line application. This will be how the initial sync is handled before letting the plugin take the lead.

Create a new repository and set it to private. It will give you the http and ssh URL for the repo. Keep the SSH URL: [email protected]:thecyberdragon/obsidian-testing.git

It also tells you the commands to do the first sync.

In your vault folder, right click > Open git bash.

This will create a .git folder in the top level of the vault (wherever you ran Git bash from).

Press the GitHub icon on the left-hand side of Obsidian to attempt to sync.
It should prompt for your user information and then should work indefinitely every button press. If not, then set up a SSH key and try to use that instead.

Note: I’ve had no success using SSH in the plugin, only the HTTPS URL.

If you try to push files manually to the remote private repo without a SSH key, you’ll get this error:

Setting up SSH to sync using Git

I use the Git cmd line method for sycning documents when I’m at work or on my laptop. It has been a challenge, but it has been invaluable for learning Git and GitHub.

This is done via your GitHub account, this isn’t set in the repository itself.

GitHub have documentation on creating SSH keys here:

https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent

Open git bash that you installed and type:

It will ask where you want to save the file – just press enter to set as the default location.

Then add in your password twice and you’ll get back your SSH key.

After the files are saved, go to the location that Git said it saved the keys. Open up the file ending in .pub -> this contains the public key. Copy the contents of this.

Back on the GitHub website -> At the top right where your profile picture is, click that and go to Settings, then go to SSH and GPG keys.

Create a new SSH key and give it a name.

Paste in the pub key and save. You should now be able to complete your first push!

Go to the location of your vault, right click in explorer and open git bash.

Now, run the commit code:

You will be asked for a passphrase, this is the one you used when setting up the SSH key.

Check your GitHub repo – there are the files! Hooray!

You have to do this for every non-primary desktop / laptop that you want to add into this Obsidian ecosystem.

Git commands for cmd line use

All commands in this section, contextually, are written in a Git Bash window that is run from the explorer folder containing the .git folder (top level of the vault), or done in cmd using cd to set the working directory to the folder containing the .git folder.

Init = initialise a Git folder in the current directory (.git)

Branch = A term for alternate commits that can be merged into the main branch later (I don’t use this)

Pull = pulling the GitHub repository to your git folder on your PC

Push = pushing your changes to GitHub

Stash = putting aside and saving changes you’ve made locally that conflict with the GitHub repository

Remote = a keyword to use to save you typing the SSH URL every time

Stage = add files you’ve made changes to to a staging area

Commit = save the changes to the staged files and set a message to explain the changes

1 – Initial sync before starting

Providing you set the SSH URL as “origin” using git remote add as shown above.

This will pull all of the updated files to your current vault folder. If you’ve made changes in any way to files that have been updated in the remote repository (GitHub), then you will need to resolve this conflict before it can pull  your files.

If you follow the rules of the land below, then this should be an accident as you’re on top of syncing before and after making changes. In this case use the stash commands.

This will store the changes you’ve made, pull the remote repository, then delete your stashed changes, effectively forcing a sync.

A word of warning – it was a HARD reset command that caused me to wipe my GitHub repository. Use stashes. If you want to reset changes you made to a file, you can use:

2 – Checking which files have been altered and adding them

This will show all files that have changed. You can then add files one-by-one. If files or folders contain spaces then the whole file path needs to be enclosed in quotation marks.

When using cmd / git to do this, you can type part of the file / folder and press TAB to auto-complete it. Make sure you complete the beginning of the folder or file so that there could be no conflicting choices first. If you have two folders:

This_awesome_folder
This_awesomer_folder

Done type This_a + TAB or you might be unpredictable behaviour. Instead type This_awesome_ + TAB if your intention is to auto-complete to get This_awesome_folder. Additionally, it may auto-complete folders with spaces and put them in quotation marks such as “folder\folder name”- you can continue typing and auto-completing with TAB, just put the cursor before the last quotation mark.

3 – Pushing staged files to the remote repository

That should do it! Repository pulled, prepared for changes and pushed back to GitHub.

Setting up Git sync on old android models

My phone is quite old… I can’t install apps that would make sycning to my phone as easy as the Obsidian plugin, so I have to use a Git app specifically.

I found and used an app called MGit.

Click on the three dots and go to settings. Enter your username and email, then click on SSH Keys.

Then, click on the +, name it (I used the name of the repository), leave it as 2096 RSA and generate key. This will generate a SSH key which you can then go into and copy the contents of.

Secondly, in settings, go to General > Root storage location for repos. Set this to where you want your vaults to be stored on your phone. You can either do this, or leave it as it is and when adding your vault in Obsidian, add it from that default path (something like /storage/emulated/0/Android/data/com.manichord.mgit/files/repo/).

Then go to GitHub on your mobile via a browser, not the app. We need to add this to the actual repository. Sign in and go into your repository. Three dots > settings. Under security, go to Deploy Keys. Click on Add deploy key.

Name the key in the Title box, paste in the key into the Key box and click Allow write access.

Back into MGit, now.

Click on the + and put the repository URL in the remote URL box.

It should automatically add the local path as the repository name.

Important note: MGit isn’t able to differentiate between different SSH keys, meaning you can only have one repository synced in the app (because the same SSH key can’t be used twice for two separate repositories in GitHub). If you have multiple vaults, then it might be more appropriate to edit files directly in GitHub using the GitHub app, or find an alternate sync method.

MGit is pretty much a UI on top of Git. The same principles apply as with syncing using cmd or Git Bash – pull the repo before you do anything, add files you update to the staging area, commit changes with a message, push to the remote repo when you’re finished. This is all done through a mix of hold-pressing on files to add them to the staging area, and pressing the three-server icon in the top right for commits, pushes and pulls.

Setting up Git sync on newer android models

For newer Android devices, there’s a very easy way to do this. I use an app called Git Sync (the icon is an octopus).

Instead of using SSH, it uses GitHub Auth so you effectively just sign in and start using it.

Click on the + (if you have more than one sync set up, click on the three dots then +). Type in the name of the repo you want to sync. When the sync now screen appears, click on AUTH. You will have to log in to your GitHub and authorise the app.
Then select your repository name from the list. It will open your device’s directory so you can choose where to clone it and sync from. Select the folder to put it in. Done. It’s quite easy. When you want to sync, just press the button. There is an auto-sync option, but I like the control of syncing it myself.

Rules of the land

While developing this system, I’ve developed a few rules to make sure that nothing goes wrong (because it has a few times).

  • Always pull from the remote repo before writing anything
  • Always push updated documents after altering them, or keep track of the changes and do it when you finish
  • Never use git add . to add files as this will add the .obsidian folder which changes as you use obsidian, add files / folders individually.
  • Use reader mode whenever possible if you’re not updating documents to prevent Git from thinking you’ve made changes.
  • Only use the Git Sync plugin on one dedicated primary machine
  • When committing changes manually, use the naming convention: PCName YYYY-MM-DD HH:MM to help track when and where changes come from.

I use the following sync protocol on my devices:

Home desktop PC -> Sync using the Obsidian GitSync plugin
Work PC -> Sync using cmd

Laptop -> Sync using cmd

Phone -> Sync using MGit

Tablet -> Sync using Git Sync

Summary

The end result, after a bit of learning Git and GitHub, is a cross-device locally-hosted super fast and ultra-customisable knowledgebase and note-taking system that also doubles are multiple layers of recovery should anything get deleted. It took a while to learn the quirks of Git and I’ve deleted my GitHub repository at least once, but now I feel confident with my Git usage that I don’t think this is going to happen again. The next step now, is to set up a locally-hosted LLM that’s capable of reading my documents so I can chat with my knowledgebase!

Git and GitHub if you haven’t started using it in any sort of way, is quite intimidating; it has it’s own language (pull, push, stash, commit, stage, branch etc) but once you have the basics down, it’s not that difficult. For anyone on the fence, it’s very worth the effort.