What “engineering maintenance day” is, and how it keeps us sane

Author: “You should keep your dependencies up to date”
Reader: “No shit, tell me something I don’t know :)“

However, most teams I’ve worked in fall behind on maintenance over time.

Why?

Because keeping dependencies up-to-date takes time and effort. In an environment when shipping is the primary objective, maintenance takes a back seat. Sure, when the effects of outdated dependencies are apparent (something is breaking, or development is slowed down significantly), it’s easy for anyone in the organization to say “we should stay up-to-date with these things!”.

A couple of examples when:

  • The new major version of Ruby on Rails has been released and six months later, your other dependencies stop releasing updates for your old major version.
  • The new shiny version of Webpack 2 (or 3) hit the streets, and all your loaders all of a sudden stop supporting Webpack 1.

You don’t necessarily need to upgrade, but the day you do (security issues perhaps), it’s going to take you way longer to do the migrations. Compared to if you had done it regularly. The challenge is to be proactive enough that the team isn’t put in those situations.

With that in mind, what are the actions we as developers can take that ensures that the outcome never becomes a production halt? For us, the main action is Engineering Maintenance Day (EMD).

EMD occurs one day a month, per project, where we take the time to update dependencies, do risk assessment and look at our current technical debt.

We do it because we don’t want to fall behind on our dependencies and software as the pain of getting back on track has a tendency to accumulate quite heavily.

Software development these days is moving at a rapid speed and the more code you touch, the more likely you are to introduce bugs, even if your intent is to improve it. That means increased development time, testing, and therefore cost ($$$). And if it slips through all the way to a release with a medium to high-defect, everyone need to foot a huge bill.

On top of that, one of the larger macro-trends over the past decade, is that the use of third-party dependencies has exploded. For example, the open-source-JavaScript community right now is just the largest, most active open source community that has ever existed.

Check out the stats that GitHub announced last year: https://octoverse.github.com/ 

According to the statistics published by GitHub, open source JavaScript activity as measured by pull requests has doubled in the past two years. It’s more than the next two languages (Java and Python) combined.

“Open source JavaScript activity as measured by pull requests has doubled in the past two years”

Given we are heavily dependent on JavaScript at Sparta — we decided to implement something called Engineering Maintenance Day in order to ensure we minimise defects that are shipped to production.

We have three different products within our team, that each require a variety of tasks during an EMD.

API (Ruby, Ruby on Rails)

  • Heroku (Ruby version) — Are there any new major or minor releases that we could benefit from?
  • Database (Postgres) — Same as above (we recently updated to 9.5+ because we needed JSONB as a column type)
  • Gems — Go through the changelog’s on a regular basis. Use a lockfile. One of the tools you can take a look at is Gemnasium which helps automate this

Web (Node/Express)

  • Heroku (Node) — Are there any new major or minor releases that we could benefit from?
  • NPM packages — Go through the changelog’s of the updated dependencies. Lock your versions with a yarn lockfile or npm-shrinkwrap. Take a look at Greenkeeper.io which helps you automate this.

Mobile (Swift)

  • Cocoapods — Are there any new major or minor releases that we could benefit from?

Verification during an EMD

This is one of the times where proper test suites really shine, since we can update dependencies and quickly find any issues.

  • Unit tests
  • End to end tests

Forks

There will always be reasons for a fork to exist. In a larger organisation it will be hard to keep track of the different reasons. As always, it’s good to maintain a documented list of these in case the maintainer at your company decides to move on.

Example:

(Dependency name). Reason for fork: Forked because we decided to expose a private attribute. Pull request upstream has been submitted at 12/2 2017 by <Name>

We make sure to always put the forks in our own organisation’s namespace on GitHub. That way we as an organisation maintain control of them.

Locked versions

There might be reasons that some dependencies have been locked to a specific version. Perhaps it involves a larger refactoring because it exposes a new API. Either way, these needs to be documented somewhere.

If your dependency file (Gemfile, package.json, Podfile) supports comments, that would be a good place. If not, you should put it in a shared wiki of some sort.

Risk assessment

Even if we do EMD on a regular basis, we can not avoid that some work are just too big to fit into one or two days. This is why we keep a separate document where we list our risks and/or technical debt that we need to deal with at some point.

Its organised in a way that lists “Risk for <X>, because <Y>, which means that <Z>. Therefore we need to <A>”

Example:

“Risk for timeouts to occur, because the data layer (SQL queries and relations) is not optimized for the size of customers we are taking on, which means that scaling to bigger accounts will be harder. Therefore we need to spend some time on optimizing these parts”

These items can then be reviewed perhaps on a bimonthly schedule where you go through them and allocate proper time and resources to the bigger maintenance projects.


This blog post was written together with my colleague Jonas Martinsson at Sparta. Shoutout to Erik Hedberg for technical proofreading.

Migrate to Ruby on Rails 5.x using RocketPants

At Sparta, we’ve been using a Gem called RocketPants since day 1 (roughly three years now) to easier maintain, version and develop our internal API. The Gem has since then lost a decent chunk of its community and it looks like less people are using it.

I would guess that since the Rails API Gem was merged into Ruby on Rails with the 5.0 release, the need for a Gem such as RocketPants shrunk.

Anyway, I’ve been doing some work on making the Gem support Ruby on Rails 5.0 and 5.1, and it seems like some other people were stuck on the 4.x branch of Rails because they were unable to upgrade.

RocketPants with support for Ruby on Rails 5.0 (and 5.1) : https://github.com/Sutto/rocket_pants/pull/143/files

I’ve been using it myself on Ruby on Rails 5.0 for the last year without any issues, so the whole Gem feels fairly stable.

This is not a permanent solution as the Gem has lost a big part of its community and we want to avoid getting stuck on a Gem that will not be developed any further, I will follow up in another post on how we at Sparta will migrate to another solution. Most likely we will be using the built in Rails API.

In addition, my colleague Yi got the rocketpants-rpm (provides support for making RocketPants::Base work with RPM) gem up and running for Rails 5 as well. It’s available under our Github organisation here.

 

Inspect SQL queries generated by Ruby on Rails

I’m currently working on a rewrite of XCStorm, my website dedicated to crawling the web for news about cross country skiing.

Preview of the new filter system at XCStorm

I’m using Heroku for hosting and for development I’m running SQLite and for staging/production its Postgres (I know its bad practice to mix…). Anyway, I found myself in a situation where I needed to compare SQL queries generated by the ORM.

Create a file named .irbrc and put this into it:

ActiveRecord::Base.logger = Logger.new(STDOUT)

Afterwards, start your Rails console and enter your query.

The reason I’m mixing database adapters is because at this point it doesn’t matter since the data won’t be complex and its easier for my partner to setup and interact with the database.

Att skriva en egen parser och kompilator i Ruby

I Februari bestämde jag mig för att ta en titt på hur parsing och kompilatorer fungerar, jag valde att imlementera en i Ruby och här kan ni se resultatet.

Det påminner väldigt mycket om tankesättet vi hade när vi gjorde en kalkylator i Java för några år sedan i en kurs jag hade på Uppsala Universitet. Jag implementerade allt fram tills dess att jag skulle generera den slutliga koden. Kunde inte riktigt bestämma mig för vilket språk jag ville generera, ren Rubykod, Assembler eller C.

Exempelprogram

Se online på GitHub

/* PROGRAM NAME: nxx1.txt

nxx is a simple programming language that provides:
 numbers
 strings
 assignment statements
 string concatenation
 simple arithmetic operations
 print capability

comments may be enclosed in slash+asterisk .. asterisk+slash
*/
alpha = 16 ;
beta = 2 ;
resultName = "delta" ;
delta = alpha / beta ;
print "Value of " || resultName || " is: " ;
print delta ;
print "\n" ;

Här är syntaxträdet som produceras av ovanstående kod:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Here is the abstract syntax tree:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ROOT
    =
        alpha
        16
    =
        beta
        2
    =
        resultName
        "delta"
    =
        delta
        alpha
        /
        beta
    print
        "Value of "
        resultName
        " is: "
    print
        delta
    print
        "\n"

Lärdomar

Själva implementeringen av parsern, lexern och scannern gick bra. Dock märkte jag i slutet att jag saknade en del kunskap om teorin bakom, vilket gjorde det svårt att slutföra. Dessutom fick jag annat att göra så det hamnade i skymundan.

I höst kommer jag läsa en kurs i kompilatorteknik, det ska bli roligt eftersom jag har gett mig på en del av begreppen tidigare och kommer förhoppningsvis ha nytta av mina tidigare erfarenheter och på så sätt ha mer nytta av kursen.

Virtuell utvecklingsmiljö med VirtualBox

Ett av problemen jag haft under längre tid har varit att jag trivs alldeles för bra att utveckla med min MacBook. Detta för att det är så mycket enklare att manövrera i operativsystemet mha terminalen, samt att det är så smidigt att använda Git, Ruby, Sass etc. you name it.

Jag försökte för någon månad sedan att åstadkomma en liknande miljö med VirtualBox och Ubuntu, det gick åt pipan. Igår kväll så började jag nysta i det igen och fick faktiskt fart på det hela!

Mitt mål

  • Sätta upp VirtualBox
  • Installera Ubuntu
  • Använda Ubuntu som utvecklingsserver och dela kataloger mha Samba
  • Kunna SSH:a till denna genom att denna får ett internt ip samt att den ska ha internetåtkomst
  • Sätta upp Git tillsammans med mitt Github-konto
  • Installera RVM och Rails, Padrino etc.

Sätta upp VirtualBox och installera Ubuntu

Ladda ner VirtualBox och installera Ubuntu (jag valde desktop edition). Jag har en trådad uppkoppling så jag sätter min trådade anslutning som delad och noterar vilket IP denna får (192.168.56.1). Det bör även finnas en “VirtualBox Host-Only Network”-adapter i dina nätverksanslutningar, det är denna som ska ha fått ett IP som ex. 192.168.56.1.

Gå in i inställningarna för VirtualBox och Ubuntu och sätt upp två anslutningar, en för NAT och en för “Endast värd-kort” (host-only-adapter). Ändra i din fil /etc/network/interfaces till detta:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet dhcp

auto eth1
iface eth1 inet dhcp

Kör ifconfig -a och kontrollera så att du har två olika IP:n på din eth0 (min är 10.0.2.15) och eth1 (min är 192.168.56.101).

Har du fått till detta så kan du förhoppningsvis SSH:a till maskinen (förutsatt att du installerat en ssh-daemon)!

Sätta upp delning av kataloger

(Källa: Ubuntu Forums – HOWTO: Mounting SMB Shares)
Härligt, du har en fungerande Ubuntu som du nu kan starta upp och gömma i bakgrunden och sköta genom ex Putty. Eftersom jag fortfarande kommer sköta själva programmeringen i Windows-miljön så kommer jag vilja ha en katalog som är delad mellan Ubuntu och Windows. Jag skapade en katalog på Windows-datorn som jag kallade för VirtualUbuntu som jag satte som delad, sedan skapade jag en katalog på Ubuntu som jag planerar att montera denna nätverkskatalog i.

sudo mount -t cifs //192.168.56.1/VirtualUbuntu /home/mittusername/VirtualWin -o iocharset=utf8,username=mittanvandarnamn,password=mittlosenord,file_mode=0777,dir_mode=0777

Detta monterar katalogen, nu vill jag att den ska monteras varje gång jag startar upp min virtuella Ubuntu. Så gå in i /etc/fstab och ange något liknande i slutet:

//192.168.56.1/VirtualUbuntu /home/mittusername/VirtualWin   smbfs  auto,credentials=/root/.credentials,uid=1000,umask=000,user   0 0

Jag har även skapat en fil i /root/.credentials för att hålla mina uppgifter med följande innehåll:

username=your_smb_username
password=your_smb_password

Sätt sedan rättigheterna på den filen med:

sudo chmod 600 /root/.credentials

När du sedan startar om Ubuntu kan du enkelt montera med:

sudo mount -a

Sätt upp Git + Github

Installera Git med

sudo apt-get install git-core

Följ sedan guiden på Githubs hemsida om Linux-installationen.

Installera Ruby med RVM

För att kunna ha möjligheten att köra flera olika Ruby-versioner (väldigt behändigt!) och framförallt hålla dessa separerade så använder jag RVM – http://rvm.beginrescueend.com. Detta kan du installera genom att följa instruktionerna på http://rvm.beginrescueend.com/rvm/install/.

Några vettiga kommandon med exempel är:

  • rvm install ruby-1.9.2-p0 (installera 1.9.2 via RVM)
  • rvm list (listar Ruby-variationer, indikerar även vilken som är aktiv)
  • rvm use ruby-1.9.2-p0 (välj den specifika Ruby)
  • rvm package install zlib (mer om detta i nästa stycke)

Installera Rails

Jag valde att köra Ruby 1.9.2 via RVM och då stötte jag på patrull i form av gnäll på att jag inte hade zlib och openssl, en snabb sökning gav följande resultat:

Följer du dessa instruktioner så borde det gå bra att få in Rails.

Summan av kardemumman

Detta är skrivet som en liten “kom ihåg”-post till mig själv inför nästa gång jag ska genomföra detta, förhoppningsvis har någon annan nytta av det också! Lycka till!