Do we all agree yet?

Do we all agree yet?

There are three camps I have found around testing. 

The ‘tests are necessary’ camp. I don’t know about you but I am a card carrying member of this camp. To be a professional software developer I think writing tests is necessary. 

The ‘I’m not writing a real app’ people. When writing a script to get a one off ops task done then it would be a waste of time to write automated tests, if it is easier to figure out if it worked manually. I have no problem with this one. Although I am surprised by how often one off hack ends up in production for years. 

Finally, there is the ‘rock star developer’ who thinks testing will only slow them down. They are quite often happy for other, normally less senior, developers to write tests for their code. Where any bugs found can be dealt with later but should not count as real development work as the implementation is already ‘done’. I find this type of attitude more of a problem than the lack of testing. You can get business value quickly, and it can even provide value for years, with this attitude towards testing. I’ve worked on apps that were critical to the business and survived in production for years with no tests at all. But these apps are unmaintainable. They are very hard to change, either to apply bug fixes or even just to keep them up to date on the latest hardware or kernel version or app engine version. This stores up risk and costs for the business which at some point must be addressed. If you don’t intend your application, or the company you work for to be around in two years, because it will have failed, or you plan to be acquired and be out of there, then this extended prototyping approach may be a path open to you. I would still want to think very carefully about this approach because code has a habit of sticking around much longer than you thought it would. It also does not benefit from any of the advantages of TDD. But be aware, that if your company is successful and someone needs to maintain this code in the future then the company is going to pay a high price for your decision.

All of this was meant to be a rebuttal to the obvious arguments that I have heard over and over again about why this team is special, why I am a better programmer than you have met before, why my application will be fine without tests. But really the best reason to write tests is that TDD is a better way of writing software and you get all the tests as part of that. Testing first allows you keep on track and only build what you need, it concentrates the mind. Writing tests before the implementation guides your design to be easily testable, but also to be cleaner. With TDD it is hard to get low test coverage and so future developers, including future you, will have the confidence in their changes by just running the test suite and seeing green. The pull to move faster, that this time is different, that it’s too hard to test this thing is strong. TDDing is the professional way to write software to last. 

Professionalism

We are not professionals. Doctors, lawyers, civil engineers. They are professionals. In comparison, the development community are a bunch of monkeys. We are banging on keyboards, hoping to knock out the works of Shakespeare.

There isn’t consensus on what the word professionalism even means. But like pornography, we all know it when we see it. Some common themes seem to be:

  • considerable training, over and above most jobs
  • vocational
  • continued training for life
  • entrance exams
  • a code of ethics
  • upholding standards
  • membership of professional bodies
  • working for the public good

In exchange society grants:

  • higher pay
  • protected designation
  • autonomy on deciding what is good practice
  • self-regulation
  • respect
  • higher status

Software development hits some of these highs. We have professional bodies like the IEEE, ACM, BCS but not everyone is required to be a member to call themselves a developer. We do tend to get people who view it as a vocational career and care about the broader implication of their work in the community. But we don’t have a universally recognised code of ethics. We do not weed out bad actors effectively or rank junior to senior members consistently.

How many times have you said no to your boss when you know what they are asking you to do will end badly? Not because you don’t want to do it, or you think it is the wrong business decision, but because your professional opinion about the code tells you it will not work. Business leaders are suppose to make business decisions. If the company really thinks that after careful consideration, getting the new product out before CES is worth so much that they are willing to pay to fix all the bugs afterwards, then it can be ok to rush ahead, move fast and break things, if that’s what it takes to win. But often they haven’t understood the risks, they don’t know how much it will cost and you can see disaster coming.

I’ve asked developers I respect about this issue and most of them have stories of when they have said no. When it was just so clear that this change wasn’t a good idea that their ‘professional pride’ wouldn’t let them stand ideally by and allow it to happen. But for most people these stories are the exception. They happen so rarely that they are stories people tell to prove a point. There are far more stories that people tell of being forced into doing things that they knew would end badly but they did them anyway in order to avoid a fight with their boss. Anyone working as a developer has something to lose by fighting with their boss. A livelihood for their family, stock options, reputation, or just the loss of a comfortable situation. It’s not easy to say no and risk loosing all that. I think part of the reason we don’t push back more often as an industry, is that we aren’t sure ourselves which things we should be taking a stand on.

Compared to other industries, software development has not been around that long. We had a very slow start from Ada Lovelace in the 19th century to teams of developers in the 60s to a recent explosion of millions of developers today. But we still don’t know how to teach it that well. We don’t have an agreed upon practice of how to build software. Or an idea what our place is in the hierarchy of a company. There is a new javascript framework out each week but we still haven’t agreed on the ideal way to test. Unlike some industries we work on the insubstantial, the invisible. You can’t jump up and down on software as you might a bridge to test if it’s strong enough to drive over. People outside of the industry can’t tell how good our software is until it breaks. Most developers find it difficult to judge other’s work objectively. We have a lot to learn, that keeps changing under us, but no where near as much as doctors have to contend with.

So what do we need to do? I think it is going to be a long journey. I cited doctors, lawyers and civil engineers at the start of this post as professionals. They have all been practicing a few thousand years so we have some way to go yet. But there are a few signs that we might be heading in the right direction.

We have a lot of passionate people who consider it a vocation. We have a lot of people who view the public good as one of their main motivations. We have professional bodies who try to raise standards. If you are reading this then you obviously take continued professional development seriously.

So what next? I think if we keep pushing the boundaries of our knowledge of patterns, best practices and hold one another accountable then each day we will be taking a step in the right direction.

I have a friend who told me that when they started medical school they where told that it wasn’t a competition. Other departments grouped students in to top 10% or bottom 10% etc. to award grades. This created a zero sum game where the worse someone else did the better it was for you. Why would you help a fellow student in that situation? But here in medical school, you would fail if you didn’t help each other. There was no way that anyone would get through medical school if they didn’t have the support of other students. The course was designed such that you couldn’t complete it on your own. The advice was clear. Help each other, share knowledge and encouragement. You won’t make it alone. Developers are often quick to criticise each other, to point out the flaws in arguments because we see our profession as so logical and data driven. But that doesn’t mean we can’t help each other. If someone doesn’t know something that you do, great, that’s an opportunity to help someone and increase the total competency level of our profession. It’s also the right thing to do.

As a final note I wanted to mention that there is an argument for defending amateurism. G.K. Chesterton said “If a thing is worth doing, it is worth doing badly.” He was subverting Philip Stanhope, 4th Earl of Chesterfield who wrote “Whatever is worth doing at all, is worth doing well”, but he is often misinterpreted as meaning that any attempt is better than no attempt. In fact he was writing in his book “What’s Wrong with the World” about the importance of amateurs. He said “that the most terribly important things must be left to ordinary men themselves – the mating of the sexes, the rearing of the young, the laws of the state”. Professionals might have many years of experience, they may be able to draw on the research of other professionals in their field. But the amateur does these things out of love, not for money.

Many people get into the world of development, not through formal training but through enthusiasm. They are self taught ‘hackers’. These are the people who stay up all night to solve that one last problem. Without these people the Open Source community would not exist. Companies have now pilled on because of how popular it has become, but I don’t believe they would have started what we see today as Open Source on their own. With amateurs we get passion in our industry. We get nonconformity. We can avoid gatekeepers and broaden the demographics of developers. Becoming more professional in the work place is surely a good thing, but lets not forget about the amateurs.

The Biggest Reason to Test

There are many reasons to write tests for our software. Validation. Verification. Non-functional requirements testing. Catching mistakes. Identifying edge cases. Documenting defects. Being the first user of our own interface. But I think one reason to test ranks above all the others. Maintainability.

I have seen and heard about many projects that start off promisingly. Features are delivered quickly by a skilled development team. Managers and Executives realise they have a great team that can produce the goods and decide to double down on the product. The team is encouraged to move quicker, to deliver features in shorter and shorter time frames. To take on more ambitious goals. In order to keep up with this ever more demanding pace, the development team cut anything that is not required by the current push for features. This normally includes thorough automated testing.

The problem occurs sometime later, normally about 2 years into the project, when the managers realise it now takes their team much longer to implement new features. They have the same number of developers as before, they are working the same number of hours. So what’s wrong? The code has become so messy and unmaintainable that no one is confident making changes, so everything takes much longer. There is then much handwringing and pointing of fingers as to whose fault this lack of productivity is. Then the project is cancelled.

This is a build up of technical debt. By technical debt I mean: the business owes a debt to the development team. By borrowing from their future selves the business has decided that moving quickly now is worth more than it will cost to pay back the loan later, making more money overall. This is the same calculation businesses make when taking out monetary debt. When the business makes this decision consciously then it is their decision to make, and it will either work out or not as they press up against market forces. But when this decision is taken lightly, with either no thought to the consequences or when non-developers are dismissive of the implications being warned of by the development team, then the whole project can fail.

By building in to your process thorough automated testing with aggressive refactoring you can attempt to hold off this fate. A good suite of automated tests, that can be run easily by the whole development team allows you to add new features and refactor with confidence by showing that your changes didn’t break anything. By being able to run tests quickly and repeatably developers need to keep much less of the program in their heads when making changes, so that they can concentrate on the important changes they are trying to make. I don’t think I can over state this, tests are key to maintainability. If you don’t want your program or your business to be around in a years time, then you can take all the risks you want to move quickly and break things. If you want to maintain a steady pace of development without a gradual slow down, then you need to invest in testing.

Most professional developers now seem to now agree that tests are necessary. But one of the marks of a professional is that we stand up for what we think is in the best interests of our client. We need to do more of this.

HTML programmer is not a career

HTML programmer used to be a job. In the early days of the world wide web there weren’t many people who understood how to make a web site. How to stand up a LAMP server, or even customise a landing page from an ISP. This meant that you could learn a bit of HTML and pick up a few basic design concepts, right click – view source on any pages you liked the look of and copy and paste your way to a new site.

As a software developer you have heard of other people worrying about their jobs being automated. Typist is no longer a job. Taxi drivers are worried about Uber and autonomous vehicles. Factory work is going the way of mining with more and more automation in factories. Not everyone can re-train as a brain surgeon.

But you’re a programmer, so life is good.

Your skills are in demand and you are making a decent wage. More than a lot of people. But if you think you are immune to this problem, you are dead wrong.

Now HTML is just one of the many skills you need to be able to be a full stack developer.

Back-end language, front-end language. Styling, design, test automation, agile, DevOps. A modern language that shows you know what is cool in the industry. A portfolio of open source work to show you give back to the community and are not all about money, and how you work on code all the time even when not at work because you love to code so much.

Designers have 99 designs to contend with, we have Upwork (now including oDesk and Elance), and freelancer.com where anyone can find a coder on short notice.

You are not a commodity and shouldn’t be treated like one. But the world isn’t fair and no one is going to give you more than the market price for your skills.

As a developer  your biggest asset is your ability to learn new things. To solve new problems. To be creative. Progress can be held back but it can not be stopped. You can either find a hole to hide in and ride out your current skills until retirement, or you can keep learning.

As a developer you have a huge advantage over most people when dealing with this problem. Part of your training, either school or on the job, has been to solve problems you’ve never seen before. To learn new things quickly and to apply that knowledge to complex problems. By learning the underlying fundamentals of programming as well as the practical skills to apply that knowledge you have already started to build up the skills you need to achieve bigger and better things. You are in an industry where you can move up the value chain to solving wider business problems and that’s where the money really is.

This is where we can help. DistributedDeveloper can show you the skills you need, the table stakes you need to stay relevant, and the areas where you can gain additional skills to stay ahead of the competition.

Getting started with Ansible

Knowledge level: Basic *nix skills.
Pre-reqs: 2 machines, one to run *nix commands on, and one remote server / VM to install the app on to.

Background

Ansible is a configuration management and orchestration tool that helps you to provision, deploy, configure and update servers.

It allows you to automated deploys without requiring a client running on each server.

This is a simple guide to get you up and running quickly with Ansible. All you need two Ubuntu machines (they can be VMs) and some basic knowledge of running linux commands. The first machine is the controller machine, this is the machine we will execute the Ansible commands on. You will also need a second machine to practise installing software on to, either a remote machine or a VM that you have the log in details for and an SSH key pair set up on. If this machine is also Ubuntu then you will be able to use the following scripts as is without modification.

I’ve been working with Ansible for a while now and I really like it. I find it much easier to follow what will be run on remote servers at run time than Chef and as I don’t know Ruby that well I like that the logic is in the configuration files.

We are going to take a simple CRUD app built in Node.js and get it running on an Ubuntu machine using Ansible.

Setup

Ansible is very simple to get started with and has great documentation, this is one of the best things about it.

To install Ansible on your local machine which Ansible calls a controller machine you can use the following method for Ubuntu 16.04:

Ansible has a personal package archive, you can add this for apt-get to use

sudo apt-add-repository ppa:ansible/ansible

and then install it with

sudo apt-get update
sudo apt-get install ansible

Config

Ansible uses a hosts file to know what servers it can talk to. It uses SSH to connect to remote servers (it can be pointed at localhost too). This means it is building on the decades of secure engineering built into SSH to protect your communication with your servers. The most basic configuration is to list the IP address or DNS resolvable name of a server you own in the hosts file

[webserver]
server.example.com ansible_connection=ssh ansible_user=ubuntu ansible_ssh_private_key_file=~/.ssh/id_rsa

Add in the hostname of your server or the ip address, the username you can login with (it will need to be in the sudoers group as we need privileged access later) and your ssh private key location.

Aside: If you don’t want to use a public/private key pair to login to your server in order to get up and running quickly, you can run the ansible-playbook command with the flag —ask-pass and provide the ansible_user’s password at stdin when prompted.

Ansible allows you to create dynamic inventory files, which involves writing scripts that output the servers ansible should talk to. We will cover these another time, but for now it is much easier to get started with a static inventory file that just lists the IP address / hostname of the server you want to connect to and the user you want to connect as.

Once we have configured Ansible to talk to a server we need to tell it what to do.

Playbooks

Ansible uses ‘playbooks’ to list actions that should be taken against target servers. These can list packages to be installed, security updates to perform, configuration to be set up in different environments and processes to be run.

We will take the example of running a simple CRUD REST app using Node.js and express. You can take a look at the app at github.com/DistributedDeveloper/examples/tree/master/apps/REST/CRUD

We will install Node.js and npm, download the source code for the REST app to the target server, install the npm packages we need and all using Ansible. We can then login and run the app.

Instructions

Download a copy of the playbook from github.com/DistributedDeveloper/examples/blob/master/apps/REST/CRUD/playbook.yml

This file is in .yml format (called yaml format). Ansible has a ton of built in modules that allow you to perform common tasks on servers like running shell commands and capturing stdout or running docker containers.

We use a few here. Take the time to have a quick look at the playbook.yml. We

Target only ‘webservers’ from our hosts file.
Install Node.js using the Ansible apt module which uses apt-get.
Install npm so we can install node packages later.
Create the directory we want to save the app in.
Checkout the demo app git repo from github.
Install the pre-req packages for our node app.

Now to configure the server.

ansible-playbook -i hosts playbook.yml —ask-become-pass

That’s it. That’s all you have to do. Try it now!

This will connect to the hosts in your hosts file, that are labelled webservers, and run all the tasks in the playbook.yml against them. It will also ask you for your remote server’s user’s password. We could have created a more complex playbook to allow us to install Node.js and npm without this but for now this is fine.

Now log into your remote server and go to /home/ubuntu/apps

You should see the checked out git repo of a sample REST API app.

cd into the directory and run

npm start

You can now go to

http://remote.server.com/ping

and see your wonderful app running.

Next time we will see how to run the app using Ansible in a docker container so we don’t have to login to run it.