Chief Yak Shaver

Here's a bunch of completely random stuff I learned these past few days, thanks to a devilishly borked up bork I fell into while trying to do what I thought was an easy task:

Just set up CI with some super basic tests on a Rails app I'm writing.

So-called "easy task" ate up my whole week, as I shaved yaks in a variety of domains I have [at present] a very limited understanding of:

  • Ruby
  • Ruby environments
  • Ruby on Rails
  • Docker
  • Continuous integration testing, aka CI
  • Gitlab's implementation of CI, via Docker

I am presuming that my specific problem will impact no one ever, and that its content will be of limited interest to most everyone, except maybe --

xkcd

xkcd: wisdom of the ancients

But what I would like to take you on is my, ahem, emotional journey here, people, because it really embodied one of the things I adore - and sometimes despair at - in tech: tenacity, grit, self-teaching, focus. AKA shaving the yak.

yak shaving (uncountable) - Noun

Any apparently useless activity which, by allowing you to overcome intermediate difficulties, allows you to solve a larger problem.

I was doing a bit of yak shaving this morning, and it looks like it might have paid off.

wiktionary's entry on yak shaving

The thing about shaving yaks is that, crucially, it feels useless while you're doing it. You're banging your head against a wall, there are perhaps imperceptible cracks in the wall, you have a thousand StackOverflow tabs open in your browser, and you generally feel like you are WASTING YOUR DAYS. At least, that's how I felt.

But I did break through the wall!

I mean, it's still all broken, but now the pieces are comprehensible and I'll just spend tomorrow fixing them, one by one. But the main, confusing, confounding, uber-borked, non-deterministic-computing-makes-no-sense yak has been shaved. I Fixed The Thing.

So let's talk, first, briefly, about what the thing was and the fix.

Then, let us speak of the journey.

The thing that I wanted to do

Context:

  • I have a very simple, very basic Rails app.
  • It is on GitLab.
  • It has 2 very simple, very basic tests.
  • I ran the rspec testing framework locally. It was fine.
  • I thought I should use GitLab's fancy continuous integration uh... integration and add a .gitlab-ci.yml file which basically said: hey GitLab, gimme Ruby version 2.5.1, bundle install from my Gemfile.lock, and let's run that basic rspec, eh?

Reader, it borked on step 2 (bundle install).

The thing that stopped me

The way it borked was dastardly and weird. Basically,

$ gem install bundler -v $(grep -A 1 "^BUNDLED\ WITH$" Gemfile.lock | awk 'NR == 2 { print $1 }')
Successfully installed bundler-2.0.1
1 gem installed
$ gem list bundler
bundler (2.0.1, 1.16.6, default: 1.16.2)
$ ruby -v
ruby 2.5.1p57 (2018-03-29 revision 63029) [x86_64-linux]
$ bundle _2.0.1_ install
/usr/local/lib/ruby/site_ruby/2.5.0/rubygems.rb:289:in `find_spec_for_exe': Could not find 'bundler' (1.16.6) required by `$BUNDLER_VERSION`. (Gem::GemNotFoundException)
To update to the lastest version installed on your system, run `bundle update --bundler`.
To install the missing version, run `gem install bundler:1.16.6`
 from /usr/local/lib/ruby/site_ruby/2.5.0/rubygems.rb:308:in `activate_bin_path'
 from /usr/local/bundle/bin/bundle:23:in `<main>'

So many questions here. SO MANY.

  • Main question: Why does it complain about bundler 1.16.6 when I (a) specifically installed version 2.0.1 (gem install blah), and (b) specifically used that version too (bundle _2.0.1_ install)? (FWIW, when I did plain bundle install, it said I needed to use Bundler >2. 🤦‍♀️)
  • Why is my site_ruby filepath pointing to 2.5.0 when I specifically pulled the 2.5.1 Ruby from Docker hub? And it says I'm using 2.5.1 when I check ruby -v?
  • Why does gem list bundler tell me I have (a) a default version of the gem (what does this mean?) and (b) that 1.16.6, the thing it later complains about, is also installed????

I was tearing my hair out here. I was googling like mad. I was facing what can only be described as a deep Zen koan-like test of my patience and resolve and sanity. I suspected something was going on with Ruby environments (that 2.5.0 vs. 2.5.1 thing? those Bundler versions?), I wanted to access the Docker container directly but - dash! - I couldn't figure out a way, because I was using the (incidentally normally very useful) gitlab-runner command line tool and what the heck does that do anyway!?! It definitely spins up a Docker container but then, well, kills it as soon as it's done. Dash and blast!

The stuff I learned along the way to fixing this

Here's some useful TILs from this terrible ordeal:

  1. gitlab-runner is a very useful way to not spam your GitLab commit histories, i.e. not embarrass yourself in front of your colleagues.
  2. Interactive web terminals are a way to pause and shell into the online CI. Useful! Not so useful locally, but oh well...
  3. You cannot unset and reset the default bundler gem. It is forbidden. Why!? How?!
  4. Bundler is kinda broken in one specific version of the Docker Ruby image. Aw, come on, man.
  5. You can replicate the above error (i.e. the one from my error log above, and mentioned in the GitHub issue in #4) by doing:
❯ docker pull ruby:2.5.1
❯ docker run -it --rm ruby:2.5.1 bash
root@7b17219881e3:/# bundle install
Could not locate Gemfile
root@7b17219881e3:/# bundle -v
Bundler version 1.16.6

So here I had a solution! That Docker ruby image wanted bundler 1.16.6, it neeeeded it, but the Gemfile.lock was insisting on 2.0.1. Fine. Let's change the Gemfile.lock file.

  1. You can change the Bundler version needed to install your app - a version which is specified in the Gemfile.lock file (created by whatever version of Bundler you have locally) - by (a) removing the version number after BUNDLED WITH (but keeping the BUNDLED WITH!) and (b) running bundle _version-you-want_ install. Thanks, aloucaslabs post! I changed this to 1.16.6 and - voila! - everyone was happy. Docker was happy. Bundler was happy. And Rails was happy. (Well, Rails was mostly happy.)

The journey

So this sort of thing - falling into a bork hole - happens once every, oh, month or two? Maybe more infrequently. It happens when your service has a weird bug-out, some unexpected behavior. Or when your data looks weird. Or when, as this time, you're building something new and have the hubris to think that "oh this'll just take a minute..."

And, boy, it is an emotional journey. Namely, it's an exercise in --

rodeo clowns

-- yes, TRUE GRIT.

Many a moment did I despair and think, "oh let's just forget setting this up, who will know" and many an excuse did I find, "oh i have more important DATA TO SCIENCE, what am i spending all these days on this for".

But I reminded myself of previous yak shaving expeditions that were similarly dire and similarly resolved, and how indeed sweet that victory was. And how much was learned! So I persevered. And also, of course, another nice thing that programming does to you is make you addicted to puzzling things out - as one part of me despaired and found reasons to give up, the other part just wanted to frickin' figure it out. AND I DID. And that boost is addictive. So: hooray!