Skip to main content
blog title image

4 minute read - Automating

That moment where you should have automated but didn't

Apr 13, 2017

TLDR; I migrated blogs over to Hugo and I didn’t automate because I was only doing it once, I should have automated because I actually migrated 450+ times (at least once per post. Find results at testerhq.com


On Risk Aversion

Because I’m slightly risk aware I have the following concerns about my main blogs:

  • blogger.com is free so at some point Google will cancel it and I will lose everything
  • wordpress uses php and a database and at somepoint someone will take advantage of a security hole and I’ll get hacked and lose everything

When I post to linkedin I have the concern that:

  • there is no RSS feed
  • linkedin might go down and I’ll lose everything

And if I ever start posting to medium or other platforms I will have the same issues.

Risk Mitigation

Therefore I decided to ‘archive’ everything into a static site so that if everything goes down I have a backup.

I chose to use Hugo for this.

Subtle Wins

This has the advantage that:

  • all posts in Hugo are written in markdown

  • I already write all my posts in markdown (and covert into html for pasting into the other platforms)

  • I experimented with my testerhq.com domain

  • I get to try out the multiple site hosting on my vidahost cloud plan

  • much harder to hack a static site (and if you do, I just re-upload)

  • I can eventually put all posts on github and have an offsite source back up as well

  • I can repurpose all the posts later

  • stuff I post to slideshare

  • instagram

  • tweets I think are useful

  • youtube videos etc.

But I didn’t Automate the migration

I looked around for tools that could easily migrate from Wordpress to Hugo and blogger to Hugo but didn’t find a lot that fit my workflow.

I thought I should really write my own but since I saw so many failed attempts out there I thought it might be too hard and take too long.

  • I really should have done

I thought that since I’ll just migrate once, that I really couldn’t justify the time to automate it.

  • I might migrate each blog once (4 of them), but there are over 450 posts, that might be worth automating.

I found a solution

My final solution was:

  • Use wordpress blogger import
  • Use a jekyll export Wordpress plugin
  • import from Jekyll to Hugo

I used a Wordpress bitnami VM and imported my blogger blogs to that, then exported to Jekyll and then imported to Hugo.

I exported my wordpress blogs directly to Jekyll and imported to Hugo.

And then the pain started.

I should probably have automated this part

I then had to check every post for mistakes during the various migration.

Sometimes it was:

  •   unconverted
  • html for iframes of slideshare and youtube messed up
  • images not convered to markdown properly
  • mad html not converted properly due to a missing tag somewhere
  • various special chars not converted properly

Pretty much every post needed to be hand checked and amended with find and replace in some form or other.

It took me ages. I suspect it would have been faster for me to automate this last part.

I did automate something

I dug out my old HTTP Testing Web Crawler and amended it to check for characters on the page this made it easy for me to check for:

  • images that hadn’t converted properly e.g. and left ![]( type chars on the html

I still have a lot of broken links to stuff I’ve linked to over the years, and I’ll slowly fix that up over time.

I almost repeated the same mistake

I was looking at an old Choose Your Own Adventure Game that I created a few years ago in Twine with my Son. And thought about converting it into more of a CYOA game book format e.g. walk out of the door (goto 8) listed on the page rather than a link.

I started hand converting it from a twine archive into markdown because I looked around and could not find any tools available that did this, and I saw a few failed attempts and I was about to say ’this is hard’.

But I knocked up a quick converter using Java and JSoup and it didn’t take long at all.

The benefit is, that because I have automated:

  • I am more likely to do it again
  • I will probably write more CYOA games
  • I learned more about JSoup
  • I have yet more code on github
  • I re-used the work I put into pandocifier

Summary

  • Sometimes we should automate, even if we are only doing a task once.