Guides

So, you buy a box of drywall screws from the hardware store. The box says drywall screws on it. The box is heavy, as if it has screws in it. When you shake it… sounds a lot like screws. But before you pay for it, do you open the box to make sure it is in fact full of drywall screws instead of wood screws or nails? If you are serious about making sure you’re getting what you paid for, you open the box and make sure it is filled with what you expect. This is QA. Quality assurance. Now imagine that instead of a box of drywall screws you’re buying a new website. Don’t you want to make sure you’re getting what you are paying for?

What Is QA?

In website development projects, QA (Quality Assurance) involves testing and verifying new functionality on a website to make sure that it behaves as expected.

Why QA?

It’s your right to make absolutely sure you get what you pay for. In truth, it’s your responsibility to exercise this right. We the developers fully expect you to do it. We can remind you of this and support the process, but only you, the client, truly knows exactly what’s to be expected of your website.

You're the client so you define excellence. We're the developers, we gladly follow your lead. We share the desire for excellence. It's reflected in our code and our system architecture choices. But you are the ultimate gate keeper of what your customers will think is an excellent website. We're counting on you.

It Goes Without Saying

When I was 12 years old, my uncle drove from his home in a town 2 days away to do a little construction project on our house. He was a master carpenter and had many years of experience in his craft. The first morning of the project, the local lumber company rolled their truck up to our house to deliver the materials for the project. I had not witnessed this type of work before so I did not know what to expect.

The delivery man handed my uncle a clipboard with an invoice to sign to document his acceptance of the delivery. To my surprise my uncle didn’t sign the invoice, but instead he jumped up onto the flatbed truck with his tools and began inspecting every single piece of wood. He got out his knife and cut open the straps binding the wood together. He didn’t grab the top boards first. He moved pieces of wood aside until he could reach the boards on the bottom of the stack. He pulled each piece out and checked its quality, measuring it, staring down its length to verify straightness and integrity. After about 5 minutes of inspecting, he jumped back down to the ground and told the delivery man to take it all back. The wood delivery was rejected.

I had never seen anything like this before. I paid close attention. My uncle did not seem to be particularly annoyed. Nor did he seem concerned about how rejecting the delivery might create an inconvenience for the driver, or make the lumber company’s day less profitable. It was clear that the only thing on my uncle’s mind was that when you pay for something, you should get ALL of what you pay for, and it should be exactly what you were expecting. It didn’t occur to him to feel ashamed or awkward about questioning the delivery, or apologetic about impacting the driver’s schedule. In fact, from my uncle’s point of view, he had a duty to verify the product. He was required to make sure all was as expected. My uncle was completely right.

This little story contains everything you need to know as a client about your QA responsibilities.

• You are responsible for the ultimate quality of the product.
• You should have (and communicate) clear expectations of what excellence looks like.
• You should feel empowered to demand excellence.
• You should take all the time you need to verify quality before signing off.
• You should be thorough.
• You should be adversarial in a professional way.
• You should avoid assumptions and insist on evidence.
• You should break big complex things into smaller simpler parts.
• You should use all of your available tools.

You are responsible for the ultimate quality of the product.

As the client, you own your website. You’re the sponsor. It’s your baby. As much as you might like to offload the problem to someone else and have them take the responsibility, it’s not possible. Only you can know what you want. Only you can know what you know.

The kind of website that you own is one that is responsible for generating revenue. Whether it’s a lead generating website or a straight-up e-commerce implementation, its job is to make money and your job is to make sure that it reliably does that job, and does it well. You were hired by your company to own and be responsible for this revenue-generating web property. You were hired to own the complexity of maintaining and optimizing it. We're the people who help you meet that responsibility and exceed its expectations.

Most websites reflect the complex business rules as well as the culture of the company that owns them. You are the agent responsible for making sure these business rules are properly translated onto the web. Your web developer cannot know the details that you know. We’re not THAT intuitive! We can guide you with good questions, but we can’t know what we don’t know, and may fail to ask you something critical.

My uncle is a great QA role model, taking the time and effort to jump up on the flatbed truck to inspect the wood, piece by piece. Only he knew what the project entailed, and how straight the boards needed to be. Only he could be the one responsible for making sure the delivery met expectations.

You should have (and communicate) clear expectations of what excellence looks like.

My uncle knew his craft very well. He was a master carpenter. Before he jumped up on the lumber truck he knew what he wanted to see. His profession was old and the standards well established. People have been taking delivery of cut lumber for centuries, and there is clear language that has evolved both for placing an order and having it fulfilled. The expectations of the lumber providers and the buyers are shared and commonly known.

On the other hand, web development is a much younger profession, with evolving standards. There are not ready-made, commonly known expectations to guide transactions as there are in the construction business. Additionally, web development can often be more technically complex than other disciplines, and finding common language with non-developers can be a challenge. Lastly, the web itself is still a work in progress. Innovations are taking place continuously, and “done” is often a moving target. Because of this, having and communicating your well-defined expectations is essential for ensuring your web developers are working to solve the right problems, and can deliver the right result for you.

The web development projects that begin with clearly defined and detailed expectations tend to end well for everyone involved. You’ve heard that house painting projects are 80% preparation? This math holds true for web development as well. Having detailed conversations, exploring and discovering options, writing detailed specifications, and documenting client expectations are all essential parts of good QA. In fact, most good QA happens in the beginning of a project before anyone has agreed to build something, as part of the project planning process. Long clear lists of features and functions are spelled out in detail. Both parties agree to the detailed expectations. Time for reviewing and revising is included in the project, along with time for testing and QA. Once deliverables are completed and ready for testing, the developers and the client can compare what was actually built to that original list of features and functions, often resulting in a surprisingly smooth QA process.

In web development, setting expectations is done jointly, in conversations between the developers and the client. The agreed-on results of the conversations are documented and signed off on by the client.

However, if you as the client leave expectations undefined, you should expect serious problems and frustrations to arise. When you expect your web developer to fill in the blanks, they will. And they will likely surprise you unintentionally.

You should feel empowered to demand excellence.

As I write this I am watching 2 large web development projects conclude almost simultaneously. One project was done for a client where virtually no one at the company asserted ownership of the project and felt empowered to demand excellence. The other project was the opposite. It was done for a client who remained intimately involved in all aspects of the project. This client reviewed and touched every detail, felt empowered to do so and asserted himself daily.

Somewhere in the middle of these two extremes is the sweet spot. But what cannot be missing from the equation is the presence of a project owner who feels empowered to demand excellence. In order to demand excellence you must know what excellence consists of for your project. The sense of empowerment here is heavily dependent on the expectation setting stages above.

In fact, clearly set and agreed-on expectations can be the only thing necessary to empower the project owner to demand excellence. Once that important preparatory stage has been completed, it can be built on to guide the entire project to success.

But it is essential that one person owns the web development project and feels confident enough to reject the work product if it is unsatisfactory. Clearly delineating the goals and deliverables for the project up front creates the conditions for the project owner to be empowered. Once empowered, everyone (especially the developers) counts on that project owner to use that power wisely and ensure that a good result is reached.

You should take all the time you need to verify quality before signing off.

My uncle didn’t hurry when he was inspecting the wood delivery. He felt empowered by his experience and clearly defined expectations to carefully review and inspect the delivery. In the business of web development there is almost always urgency. Speed is usually a huge priority. This means that there is always pressure on the developers to make the QA cycle go quickly. But remember, when the project owner has taken responsibility and has been careful to create and get sign-off on clear expectations, they have a position of power, the power to take the time needed to make sure things are as they should be.

The lumber delivery guy who came to my house that day made it clear to my uncle that he had other deliveries to make. My uncle had heard this many times before. He knew that if he let himself be rushed in his inspection, problems would go unnoticed, and could compromise the success of his project. He knew that good QA just takes the time it takes.

Think of it this way. Had my uncle taken delivery on cracked, crooked or rotten wood, the structure he built would probably not be stable, and might have collapsed on the people inside it. You can rush your QA, but that only means your customer is the one who will ultimately suffer from the lack of quality.

On the other hand, because you have done the careful preparation of setting clear expectations, your position of power permits you to demand timely delivery. Your developers agreed to a timeline and they did so as professionals because the full details of what was to be built were explored and spelled out in proper planning. You can demand on time delivery and raise a stink if it doesn't happen.

You should be thorough.

Implied in all of the above is thoroughness. You will feel pressure from all around you to conduct your QA inspections quickly, to skip over details and avoid the dark corners. Your boss will talk about deadlines, and demand that you make the project go faster. Your boss wants the new website delivered as soon as possible to get in front of a marketing deadline. Your web developer wants you to sign off soon so that they can get paid and start another project. The team helping you QA wants to move on quickly because QA can be tedious, frustrating and exhausting. Everyone has other work to do. My uncle didn’t care about the delivery driver’s other deliveries. You too are obliged to ignore all of the noise and think only of your customers. You are beholden to them.

It should be assumed that your customers will eventually find their way into every dark nook and cranny of your website as they explore it. The more traffic you have, the more customers you have, the more likely it is that someone will soon find that one little thing that you failed to test.

You should be adversarial in a professional way.

Imagine two best friends who love to play tennis together. Do they go easy on one another just because they are best friends? No. They are demanding. They don’t let up. They remain collegial, but they play hard and they play to win, and they push their opponent to do the same.

Website QA is this way. You trust your web developer, but you also take responsibility for your role and you work hard at it. You mutually set project expectations, but you also agree to do your part to enforce those expectations. You challenge each other to step up.

You may be surprised to hear this, but your web developer thrives on your critical feedback. They know they can only excel and succeed when you conduct careful, thorough, detailed QA of their work. Because website development work is so extremely detailed and labor intensive, developers often get to a point where they are so deep in it that they cannot see their work clearly anymore. Coding can be so intricate and consuming that a developer can get lost in the details and have difficulty stepping back and seeing the big picture issues.

We recently completed a project that was extremely complex. Four developers were involved in the build. Six people from the client side were involved in the QA. The deliverables had been reviewed and tested and revised and reviewed and tested again. We finally reached our target launch day and took this complicated new e-commerce system live. When the first real customer order came through the system, someone was monitoring a backend system and noticed something looked weird and wrong. It turned out that over the course of several months of reviewing the build, every single one of us had missed a serious and very costly problem with the way shipping costs were calculated under certain circumstances.

Looking back on the process, we realized why we all missed the problem. We had put too much emphasis on getting along with each other. That had become the priority for the team instead of the objective take-no-prisoners review of the details that was essential for a successful launch. We had all failed to be adversarial and ensure everyone was doing their best work.

Being adversarial can also create efficiencies. Confrontational communication can quickly cut to the heart of a problem. And this type of comportment tends to create permission for the QA team to speak directly and stick to their guns when they have found a problem. You can be completely professional while positioning yourself as an adversary. You can be courteous and kind. But you must remain clear and empowered, and advocate relentlessly for your position. Your web developer is counting on you. And so is your customer.

You should avoid assumptions and insist on evidence.

“I thought you knew the whole control panel was purple.”

Once upon a time I was at a conference for a particular CMS that many web developers used back in the day. We were talking to the CEO of the CMS company and someone asked him why the control panel was purple. The CEO had a funny look on his face. “What do you mean it’s purple? I thought it was blue.” It turns out that the CEO of the company had his monitor maladjusted. His purples were shifted to the blue spectrum. The control panel was purple, but he saw blue. No one in the whole company or outside of it had ever felt empowered to question this weird color choice. Everyone just assumed it was intended and moved forward with it.

Of all the launches we’ve overseen, the worst problems always came about due to assumptions. Every time.

In the QA phase we are always working through complex systems of functionality. We begin the process, and then quickly lose ourselves in the code, immersed in detail. A kind of blindness sets in where you can no longer see the assumptions that you and others made at the start of the project. One of the most simple remedies for this very typical and pervasive problem is to have someone outside of your team look at it. They will be seeing it exactly as it is, for the first time, with fresh eyes and no assumptions at all. Even if all you have is a sneaking intuition that something is weird, or you just know you’re very tired, it is always the right call to have one of your co-workers glance at the screen. Encourage them to ask the dumbest question they can possibly dream up. This can very often bring issues to the surface that would otherwise remain hidden until after the build goes live, when a customer finds them.

You should break big complex things into smaller simpler parts.

My uncle stood in front of the house holding the clipboard with the invoice that the driver had given him. He was staring at a large flatbed truck full of lumber. The lumber was bundled up, tied tightly, ready to drop off. The driver was in a hurry to get the invoice signed so he could leave. My family was eager to see the new project completed. There was a lot of pressure on my uncle to just sign the invoice and take the delivery so the project could get started. All of this pressure created a big tangled knot of expectation that could easily overwhelm someone, causing them to just sign the invoice to make the pressure stop. But because of his experience and expertise, my uncle wasn’t overwhelmed. He knew that the way to deal with this Gordian knot was to just cut it apart. So he put down the clipboard, climbed up on the truck, took out his knife and started cutting away the straps that bound the wood planks together. He broke the bundles apart, and deconstructed the stacks so he could examine each piece of wood completely.

This step is essential. Cut the knot. Break the big problem into smaller problems. Remember that you are a human. Your capacity for detail is limited. You can hold only so much in your mind at one time.

So if you have done a good job of preparing the project with your developer, and laid out detailed expectations, you have likely done so by grouping things and compartmentalizing functionality so that it’s more comprehensible. Now you can lean on the structure this preparation provides and test each component part of the whole in an orderly way, and be confident as you move through things that you are being thorough. Because you are tracking your progress with your lists and can see the value of your work, you can take your time with it. If someone asks you to hurry or asks you how much longer, you have an answer for them. Breaking something big into smaller pieces ultimately helps you see it more clearly as a whole, and the time you’re taking now is a necessary investment in the success of the project.

You are responsible for something big and complicated. You can only fulfill that responsibility if you can comprehend the whole thing. And you can only do this if you break it into smaller pieces. This will empower you to find flaws while they are still fixable, and ensure the excellence that your customer demands.

You should use all of your available tools.

Website QA is complicated. I’m no rocket surgeon so I don’t know how it compares with launching rockets or removing brain tumors, but I know it’s up there near the top of the list of jobs that are intricate and complex. To succeed in your QA you need the right systems and tools.

We’ve reviewed the importance of laying out detailed expectations. And you know that this isn’t something you should do alone. Get some help with this work. Figure out which planning and project management tools will best help you break out and define what’s expected and then track testing against these expectations.

Sometimes a tool as basic as a color coded spreadsheet can do the job. Sometimes you need a system as complex as Asana or Jira to get the job done. We tend to love using Trello for this purpose. It’s a project management tool that looks just like a big wall with a virtual card for each task. The way the Trello board is set up inherently lends itself to breaking big things into smaller pieces and then looking at them collectively as a whole.

Your tools should not only keep the many tasks involved in your project organized, they should also facilitate the sharing of expectations. They should support the concept of being responsible. Good project management tools help objectively assign ownership, accountability, blame and responsibility. If the tool says Jimmy was supposed to provide the shipping calculation details and Jimmy doesn’t provide them, it’s obvious.

My uncle hopped up on that lumber truck with a straight edge tool. He didn’t have to argue with the driver. The straight edge objectively proved that the wood was crooked. Let your tools do your work for you, and serve you objectively to gather and report information.

We're counting on you and you're counting on us.

As your web developer, we want to take pride in our work. We want to create excellent experiences for your customer. But we don't know your customer like you do. You own that responsibility. We seek to partner with you and support you in the very difficult ongoing work of caring for your customer, anticipating their needs and delighting them as often as possible. We invite you to question our work. QA it seriously. Trust us, but verify our excellence by testing it carefully against clearly defined expectations.

Hopefully this guide and the example of my uncle will inform and empower you, and help you to succeed at this challenging task.