Visualisation, dashboards and right metrics can help your project

I’m a visual person and that means that I’m better receiving information in a visual form than in any other (more about learning styles here). So being visual learner I firmly believe in visualisation as the best way to fight complexity, make things simpler and easier to understand and always promote it throughout my work. In this post I’m going to look at how information radiators, dashboards, visualisation and right metrics can improve organisation, help teams to be more self-organised and help build better products.

From my experience I found that it really helps to understand any complex problem if you just draw it’s context or visualise it in any other way. I particularly like Dan Roam’s bestselling books on that subject – they are very easy and interesting to read and give you lots of practical ideas on how to visualise various situations.

Being an aviation geek I can’t miss the chance to refer to aviation industry as an example:

Over time complexity of airplanes grew significantly from Wright brothers first flying machine to multi-engine piston aircraft to jet mammoths like new Airbus A380 or modern fighter jets that can’t even be controlled by pilot without the aid of computer as otherwise they will crash almost immediately. So as complexity of flying machine increased with growing number of various supporting parts and sub-systems amount of incoming information and factors pilot needed to deal with increased as well. And as we know, we humans are rubbish at multi-tasking and our brain capability is limited to focus just on a few things at the same time. Most studies show that we can effectively concentrate and deal with 7 ± 2 factors at any given time due to specifics of our short term memory. So to keep piloting safe great efforts were made to build better flight instruments that will gather all important information from all subsystems, organize it and display in effective and convenient way to the pilot.

Left to right: WWI plane, Concorde, A380

Even with all modern technologies you can display just finite amount of information – remember about our short-term memory limitations. So that’s why in any aircraft you can see only 6-7 main flight instruments are placed in pilot’s primary focus.

Same primary instruments on the old dashboard (left) and new HUD (right)

It’s also very important how information is displayed. To make most out of it you need to process raw data and represent it in meaningful way. On the image above modern HUD shows the same information as one on the left but it visualises it much better (notice little circle in the top left that shows direction to the next waypoint).

Alright, we done with short detour to aviation industry and I hope that I managed to illustrate my point that visualisation helps to tame complexity beast. There also many other examples like recent rise of infographics and just look at this visualisation of US budget – that is lots of data on one easy to understand image.

Let’s go back to software development. With increasing adoption of agile and lean practices and techniques we start to see more visualisation coming to help plan and guide projects. All those Kanban boards with workflow visualisation, value stream mapping diagrams and project metric reports help a lot. But what I see quite often in teams I worked with is that not many go beyond basic Kanban board with 3 lanes like “To Do”, “In Progress” and “Done” and burn chart. It’s often (but not always) better than nothing yet I’m sure that we can do much better with more effective and sophisticated dashboards.

In one of my previous posts I wrote about self-organising teams and that to build amazing products you need amazing teams (I tend to believe that product is reflection of the team that built it) and only self-organised team that is not micro-managed but rather guided and empowered and understand what it’s doing and why is truly amazing. I think that wisely mastered dashboard/information radiator can help with that. Sadly though that is often overlooked.

“What gets measured gets improved” – that quote is not always true but at least what gets measured is definitely gets attention. So you need to chose carefully when designing your information radiator what it will display. Showing right information will focus everyone in the team on the right things but showing wrong things will enforce wrong behaviour (e.g. testers rewarded for number of defects they opened). I never saw company-wide or programme-wide metrics to be displayed on the project dashboard and that is huge mistake in my opinion.

One of the core Lean principles is to optimize the whole and to do that we need to measure the whole. To empowered the team we need to give it more information and better context for things they are doing so sharing company/programme-wide metrics with the project team and displaying how project and every individual affects and contributes to them can make very positive impact.

How often you saw those pointless company share price displays in corporate lobby halls or on corporate intranet portals. I bet quite often. Now think for a moment if that is what displayed everywhere as one of the main metrics of how well company performs what do you think will be focus of executives, managers and other decision makers in that company? Most probably they will be focused on the money and stock prices first and delighted customers and good products last (hopefully not but odds are that they will).

But hey! Isn’t everyone talking at the moment that we should be focusing on customers, experiences and better products first and money will came as a result? Check out the Stoos Network, those are the people focused to find better ways than just focusing on the stock price.

Once you start focusing on the money it’s race to the bottom, thinking how to make product cheaper and cheaper you’ll probably find yourself with very low margin and shrinked profits. And what is important that growing middle class (look at China) doesn’t want to buy just cheap products, they want better products! People have money and they want to spend those money on something what works well, looks good and of high quality and not something made of cheap materials with poor ergonomics and will break shortly after purchase but costs less.

I really liked dashboard I saw at the British Airways main lobby hall – on a big screen it there was percentage of flights that departed and arrived on time and overal customer satisfaction index and how it changed comparing to the previous period.

In my opinion that is much much better than a stock price and every morning when you go to the office you see those numbers and don’t know how others but I quite often found myself thinking how can I make a difference and improve it.

On one project I worked at we were overhauling one of the key self-service pages of internet portal. In addition to standard Scrum/Project metrics we had metric that was showing size of the page in kilobytes and average time it takes to load and process in the customer’s web browser. Monitoring and improving upon those metrics in addition to others really helped us to improve customer’s experience and that’s what the whole project was about. Showing those metrics to every team member helped keep team focused and share that overall vision of why we are doing the project in the first place and how team’s work affects customers.

Alright, said enough about importance of the right metrics and effective dashboards, now it’s time to talk how to build one.

Guys at Atlassian did Wallboards Contest some time ago to collect various ideas and examples how different teams approach task of displaying important information effectively. It was great contest with lost of interesting examples (87 entries recorded) and I encourage you to check results here.

For many years most of Kanban boards were physical and made with tape, index cards and stickers and put on the walls, whiteboards and windows all over the offices. It was good for the beginning but physical board has many limitations and doesn’t really work well with distributed teams. The contest winner,  Vodafone Web Team solved that problem to some extent with RFID tags, projector, label printer and webcams. Here’s their video:

They use RFID readers attached to the board and RFID markers on the cards to track and update Jira as they move physical cards on the board. Projector is displaying burndown chart over the board and every time new task is added to the jira related card is automatically printed on the label printer attached to the board. How cool is that!

I had an opportunity to talk to the Greenhopper  team members at Atlassian about other ways to integrate physical boards with software solutions and they told me that some enthusiasts are trying to utilise Microsoft Kinnect controller to interact with the virtual board, solution is not production-ready yet but I expect it to be really good way to solve that problem eventually.

So when building dashboard think what to display and how it will affect team and project direction. Try to convey project, programme, company vision through those metrics and data that is displayed.

Since dashboard has limited surface don’t display what is not critical but when something bad does happen show it and make sure that everyone pays attention to that.

I also like that idea with dashboard backlighting to attract attention when it’s especially needed.

If you need to show more metrics than one screen can’t accomodate – make virtual screens and rotate them.

One metric that I like and use sometimes in retrospectives is drama-level or team happiness index – number from 1 to 10 or from 1 to 5 that shows how good is the team rated by it’s team members. Why don’t collect that metric on a regular basis and display on a dashboard?

I really like idea of Value, Flow and Quality as sort of three essential dimensions of every project or activity (read more about that here) and I’m going to review metrics from that perspective:

From Quality perspective I would monitor number of P0 and P1 defects found in UAT and production and average time to fix those defects. Application performance metrics after each deployment (e.g. if page load time or operation time increased after new deployment then we need to look at it). Frequency of the broken build is helpful and percentage of the code coverage so loved by many is good but don’t be fooled as it not always represents real picture as tests could exercise the code but not assert results properly showing high percentage but not so great assurance.

To gauge Flow I would monitor bottlenecks and WIP limits in the workflow. Size of code commits and frequency of those commits. Also good to keep an eye on cycle times of operations such as deployment and test execution.

Value probably is hardest one and the least monitored but in fact it is the most important one. The reason why it’s least monitored is due to it’s complexity and not always straightforward how and what metrics to collect. But I’m sure it’s solvable if some extra effort applied to think about it. I’d monitor things like conversion rates, sales/revenue/profits, downloads count, customer satisfaction and engagement, usability. And I’d definitely share that with the project team!

If you dashboard is accessible via web make it interactive and possible to drilldown to the details of each metric to better understand it.

And lastly apart from making it useful don’t forget to make it nice looking and fun so that it will have more chances to stay around and keep providing value. Countdown timer till the next pub crawl or something similar sounds good to me to be put on the dashboard.

If you have an example of the dashboard or metric(s) that you are proud of and want to share or any ideas good ideas in that area please add it as a comment.

Shameless Plug :-)

If you are in Australia and even better if you are in Sydney and want to improve your software development and delivery processes and build better products  – let me know and I’d be happy to help you out!

Updates from Atlassian

I spent whole day yesterday discovering new cool stuff on Atlassian website and it was definitely worth it.

Firstly I just found out that I missed release of new amazing piece of software that I believe would be important to the test community as much as Selenium. Software is called Bonfire and it’s extension to the browser (all major browsers supported) that helps you do exploratory testing and let’s you fill in bugs right inside the browser without changing context! Estimated effort saving is about 16% but there also many more positive side effects such as more pleasure from doing testing.

Google released alpha version of very similar tool called BITE and I had a chance to play with it – concept is very good but google’s version is at very early stage right now so it’s not really possible to use it in production while Atlassian’s Bonfire is commercial-grade finished product. Can’t wait to try it in real life and hopefully that will happen very soon.


Another thing that I discovered is 13-post long compilation of thoughts, reviews and ideas about testing and test industry in general. It’s not technical and rather philosophical with thoughts about exploratory testing, role and future of the test engineer and so on. I encourage you to take a look at it as it’s good food for thought.

Teams and Motivation

I was recently thinking about teams and motivation and by coincidence came across few interesting bits and pieces on that topic that I wanted to share.

First one is from the book I read recently about how Google tests software:

Pirate Leadership by Jason Arbon – an excerpt from the book “How Google Tests Software”

A Pirate ship is a useful analogy for managing test engineering teams at Google. Specifically, the Test organization is a world where engineers are by nature very questioning, wanting conclusive proof, and constantly measuring their lead and manger’s directives. One of the key aspects we interview for is being self-starter and self-directed—so how do you manage these folks?

The answer is much like I imagine the captain of a pirate ship maintains order. The truth is the captain cannot ‘manage’ the ship through brute force or fear as he is outnumbered, and everyone is armed to the teeth with technical talent and other offers for work. He also cannot manage through gold alone, as the Pirates often have more than they need for sustenance, and what truly drives them is the Pirate Life and the excitement of seeing what they can capture next. Mutiny is always a very real possibility too, as Google’s organizations are very dynamic, and folks are even encouraged to move teams frequently. If the ship isn’t finding lots of treasure, or that fun of a place to work, engineering “pirates” will get step off the ship at the next port and not return for the when it’s time to sail.

Be an engineering leader means being a Pirate engineer yourself and knowing just a bit more about what is on the horizon, which ships are sailing nearby and what treasure they might hold. Lead through technical vision, and promises of exciting technical adventures and interesting ports of call. You always sleep with one eye open as an Engineering Manager at Google!


Another one is a recent post in Value, Flow and Quality blog where research team challenged Daniel Pink‘s ideas on what motivates us. It’s definitely good to watch Daniel’s RSA talk and read opposite point of view on that topic.

For any team leader aiming for success it’s critically important to have good understanding what motivates others and how to motivate team members. Time of direct orders and authority is gone and pretty useless in creative field of work. You find yourself actively managing someone or giving direct order and using your authority as the last available option in your toolbox – bad news for you as most probably you are failed.

I’m firm believer in the concept of empowering team members and shifting more decision making and control to the front edge, to the people performing actual tasks (developers, testers, designers). I know that defence forces of various countries invested huge amounts of money and effort in research on that topic and it is one of the key concepts of a modern warfare.

US Defence Force has dedicated research program called CCRP (Command and Control Research Program) and they are issued number of publicly available books and papers on that topic. So there’s great opportunity for a cross-pollination from Defence into more peaceful areas.

How Google Tests Software

Just finished reading rough cuts of new amazing book from Google’s test experts called “How Google Tests Software”. It is available from Safari Books Online and I can’t really find enough words to describe how good and valuable that book is!

About half of the book is quite technical and focused on test automation, continuous integration and delivery, tools and techniques. Some of them quite radical and thought provoking – made me rethink my position towards Record and Reply test automation concept. I also enjoyed idea with Quality Bots and can’t stop thinking how that could revolutionarise regression testing. Also was quite impressed with BITE tool – this one really takes bug-related workflows to new higher level and should make everyone’s life so much easier – definitely will give it a try on my next gig.

Second part of the book is not technical and has lots of interviews and real examples of how various teams at google implemented change (love the approach they took – do internal advertising campaign, make it hype, introduce nice little tokens/badges to let people stand out from the general crowd), introduced new concepts and how they failed and succeeded.

So even if you are not technical and you are not so much into testing you still can find plenty of useful information about leading and implementing change.

Those who follow Google Testing Blog and GTAC conferences will find that about 70-80% of the book was already covered either in the blog or on a conference but I would still buy it for remaining 20-30% and just to have all in one place.

Thoughts on test automation. Part 2 – practical.

In this second part I’ll describe delivery process I set up for my startup project Aviacamp. It’s more hands on, practical example of the concepts I outlined in the first part.

About the Project

Aviacamp is the community and marketplace website for aviators that I started developing few months ago. It uses Ruby on Rails v3 as application platform and MongoDB for persistence. I am the only person behind the project doing whole range of activities including design, development, testing and support.

I chose Ruby on Rails mainly due to it’s rapid development philosophy, easy to understand and highly flexible notation (so you can keep your code clean and beautiful) and it’s large and very active development community. Another significant factor was it’s lightweighteness – you don’t need much hardware resources to run it as opposite to Java-based solutions.

Quick note: Even that my project is in Ruby on Rails all concepts I’m describing here are applicable to most of the modern development languages and versions of mentioned tools either exist for those languages or alternative is available.

MongoDB was chosen mainly because of my desire to try something new and NoSQL is sort of hype at the moment so I thought why not. So far I’m extremely happy with that – it’s scalable, fast, flexible and has excellent support for RoR via mongoid gem.

This project fulfils two purposes at the moment – it satisfies my passion and desire for building things and it is an excellent sandbox for experiments, both technical and social and provides great learning experience.

When I started it I tried to apply all the best principles, practices and techniques I was teaching my clients past few years. I’m really happy with the results and to be honest I’m very proud of it. It doesn’t mean that it’s perfect and I see a lot of room for improvement and definitely wouldn’t stop improving it.

Want to share some quick stats from the project to give you better idea of how good it is :-)

It is not a big project, at the moment of this writing it had 3076 lines of production Ruby code (excluding JavaScript, HTML and CSS) and 10698 lines of test automation code (1074 lines are for end-to-end/UI tests). So as you can see it’s about 1:3.5 ratio between production code and test code.

Over the past 3 months when project was in live with average 150 people visiting it every day it had less then 10 defects and all of them were very minor or cosmetic.

On average I’m doing one deployment to production every 3-4 days but on very busy day it can be up to 3 deployments per day.

Complete test cycle takes 20 to 30 minutes depending on the size of change (for minor feature it’s 15 minutes for major feature like events section it’s roughly 30 minutes). Most of the tests are automated – there are 1251 behind UI tests that run in 3.5 minutes and 225 end-to-end UI-level scenarios with 2152 steps that execute in 14 minutes. I usually spend very limited time doing manual testing – mostly it’s exploratory-type activity when I’m trying some edge case and weird scenarios to try to break or crash the system – it doesn’t take more than 5 minutes for every release and up to 15-30 minutes for new major feature.

Development cycle varies with each feature but generally it’s 1-2 weeks for each major feature like events, rentals, posts.

Alright, I hope that was enough talking to make you interested to know more about how exactly it could be done. Continue reading

Thoughts on test automation. Part 1 – theoretical.

Second part much more practical and with insights from my startup is available here.

Imagine a project where high quality software is delivered regularly and repeatedly. Project where test engineers are experts in product requirements and spend their time not routinely executing same manual test scenarios over and over again as if they are robots but instead they check non-trivial scenarios and perform exploratory testing to verify that product can handle wide range of situations.

Imagine a project where developers can change and improve codebase without fear of breaking anything or introducing defects because they know that they will get feedback on their changes in seconds or minutes and will know exactly where problem is if something breaks.

Project where everyone in the team actively participates in requirements analysis, where everyone knows and understands “What?” system should do and “Why?” it should do it.

Project where it’s absolutely clear what progress team made at any point in time.

If you think that such kind of project does not exist in a real world and if you are curios to learn how you can make your project be like that – below you will find key principles and practices I’m using to make my projects be like that. Continue reading