Measuring design debt

In this piece, we’ll dive into the topic of measuring design debt. I’ll walk you through the qualitative and quantitative measurements and share how to put them into practice based on my experiences. You’ll benefit from this article the most if you find yourself struggling to pay off design debt in your organization or getting buy-in for this initiative. Enjoy!

Every product, purposefully or not, starts with design. This is when the domino effect of both good and bad events begins.

Why should you measure design debt?

Measuring design debt requires a lot of resources. In addition, the process needs to happen in a perpetual state to understand the full context, guide in the right direction and help achieve the desired outcomes. It’s natural to ask: what’s the purpose of all that effort? Here are the main reasons why measuring design debt is necessary:

• Establish a point of reference

Having common understanding about design debt and finding out how much of it your organisation has accumulated over time is necessary to make decisions now and objectively assess any initiatives in the future.

Making it clear where your company is currently is a good first step to define and measure the goals.

• Understand the impact on business
  Customer satisfaction, rate of adoption and team velocity have a big impact on the business. They all can be affected by design debt. Measuring it and making a clear connection between (not-so-tangible) design debt to (very tangible) customer complaints or lost sales opportunities will paint a clear picture of the problem for your whole team.
• Understand the impact on culture and people
  Design debt can negatively influence the morale of your team. Smart people get discouraged when they cannot work on meaningful problems. If making minute changes takes disproportionally too much effort and results are mediocre, your team members can burn out quickly. Linking those symptoms to design debt can be meaningful for building a healthy organization.
• Assess the urgency
  During planning, it’s necessary to take a critical and honest look at the balance between efforts spent on managing design debt and other activities like developing new and existing features, investing in better technology, etc.
  
  Paying off design debt always makes more sense in the long run.

For the short-term planning, take all the trade-offs into consideration.

List all the work that will not happen because of paying off the design debt and see how it affects your contribution to the company goals. You’ll probably need to adjust the scope of design-debt-related work to still be able to deliver urgent projects.

• Track progress
  It’s important to regularly check if you’re moving in the right direction and if your efforts are actually pushing the company forward. Measuring design debt systematically provides a credible picture of the progress.

You need to determine if paying off design debt has as much of a positive impact on the product and the organization as you’ve foreseen initially.

If effects are misaligned with your projections, it’s a signal to course correct.

• Define the acceptable level of design debt
  Measuring design debt over time lets you understand how much your organization can handle. Observe:

• what amount of debt is not disturbing processes and delivery
• when your team feels comfortable with the state of the system

This will inform your decisions about how much design debt you can take on and use consciously to your advantage in the future.

How to measure design debt?

Reading the signals, gathering data, analyzing it over time and finding your limits are necessary to truly assess the design debt. This process takes some experimentation and experience to establish. It is also unique for each product and organization.

I’ll distinguish between the qualitative and quantitative methods of performing design debt measurement. They are both important but serve different purposes:

• Qualitative methods let us understand the root cause of design debt and the high-level context
• Quantitative methods are used to track progress and calculate impact.

Qualitative measurement

Qualitative measurement focuses on gathering non-numerical information that let you:

• gain a general understanding of the amount of design debt and the issues caused by it
• find origins of the problems
• discover and examine the occurring patterns
  

The most effective methods of performing qualitative measurement differ depending on the type of design debt:

UX design debt

This type of debt has a direct connection to the business goals because of its impact on adoption rate, discoverability and general customer satisfaction.

How: gathering primary and secondary user feedback

• Usability testing
  
  ‍Discovering design debt should be a secondary goal for all usability testing sessions. In addition to testing the ongoing design work, regularly test the main flows of the application with both new and existing users. It’d let you identify the areas of the product that have accumulated a lot of design debt.When performing a usability test for a particular path, look out for feedback about other parts of the product as well. Take note of all the smaller issues and pay close attention to the general thoughts testers share with you.

There is a lot of value in side topics, especially if users bring them up unprompted!

In order not to lose track of the additional discoveries, create a process of cataloging, sharing and reviewing them with your team. If possible, point out the connection to ongoing discussions and projects.

Tools like GitHub or Productboard come useful for storing insights from usability tests and prioritize problems based on how validated they are. The important thing is to ensure visibility for team members, especially other Designers and PMs.

• Collaboration with other departments
  
  ‍To collect even more insights, ask Customer Support to pass repeating feedback to you, talk to Sales about what customers are most curious about and what are the main dealbreakers, contact the Customer Engineering team to learn about the learning curve and the most frequent feature requests. This knowledge, in combination with other pieces of feedback, will let better you understand the big picture of your UX design debt.
  
  *Read more about the product debt and measuring UX debt in A practical guide to solving product debt by Laurent Grima.

Visual design debt

Analyzing visual inconsistencies in the product requires a cross-team effort to get complete and credible results.

How: design and engineering evaluation, subtle signals from users

• Visual evaluation

Tracking visual inconsistencies is done best by people who are the closest to the product.

Design and engineering should collaborate on analyzing visual discrepancies. Based on their work, produce a comprehensive report and use it to prioritize. Focus on the most problematic and the most impactful areas first. Keep the report up-to-date to measure the volume of visual debt over time and track the progress of your work.

• External feedback
  
  ‍You can cross-reference the report with external feedback: signals from both existing and potential users and customers that come up in conversations and during usability testing. Look out for phrases like:‘
  
  This … doesn’t feel like your product’
  ‘This … is the old version, right?’
  ‘This … feels outdated, are you still supporting it?’
  
  ‍Those sentences will usually be followed by:
  
  'It’s hard to say what it is exactly but something just doesn’t feel right…'
  
  ‍as the majority of users struggle to name or describe visual inconsistencies.
  
  Based on the information from two sources (internal analysis and external feedback) you’ll gain a perspective on which areas actually require your team’s immediate attention.

Operational design debt

Understanding the shortcomings of underlying structure and processes requires listening to your team carefully, gathering feedback about design organization within your company and, if you are a manager, noticing clues from your reports during 1on1s.

How: surveys, interviews, 1on1s

• Team satisfaction surveys
  
  ‍Conducting team satisfaction surveys is a common practice in many organizations. Usually, employees rate areas like wellbeing, professional growth, relationships, or company culture. General surveys are a good source of knowledge about the sentiment in the organization but they are not specific enough to examine operational design debt properly.
  
  Adding the following questions to the surveys gives an opportunity to collect more appropriate insights:
  
  ‍Were you able to do your best work over the last … months? (Scale 1–5)
  What are the main obstacles in your everyday work that stop you from doing your best work?
  If you could change one thing about our processes, what would that be?
  
  ‍Responses to the question about obstacles in everyday work expose the problems that people encounter. The score of ability to do the best work indicates how severe these issues are. The third question asks directly about processes. It provides you with both problems and potential solutions. They may be treated as suggestions when building the strategy for paying off operational design debt.
  
  I recommend leaving the second and third questions completely open. Offering multiple answers or examples would imply the direction unnecessarily.
  
  After analyzing the responses:
  

• list the operational problems, their severity and scale
• list the solutions suggested by the team

All of the above should be provided in two contexts: internal for the design team and external for the design-engineering-product group.

• Interviews, 1on1s
  
  ‍The next step in examining operational design debt is talking to designers and other team members who work with the design org on a daily basis. To avoid bias, interviews are best done as 1on1s.
  
  ‍Ask about details of the internal and cross-team collaboration and try to learn about how the design team is perceived.
  
  ‍Based on cross-analysis of the information from surveys and 1on1s, it is usually evident which processes need immediate attention. All the results should be stored to give a better understanding of the impact of changes over time. Given that employee satisfaction surveys are typically performed every 3–6 months, you can add your questions to them and sync the design-debt-focused interviews with this cadence.
  
  

Quantitative measurement

It’s time to focus on gathering numerical data. As...

there is no single unit to express the amount of design debt,

the quantitative measurement is based on a group of metrics affected by and connected to it.

Please, note that I perceive the quantitative methods as secondary. Their analysis should be heavily informed by the context gained during the qualitative research.

UX design debt

How: observe key metrics

You can analyze metrics like DAU and MAU (daily/monthly active users), bounce rate, customer retention rate, NPS (Net Promoter Score), CSAT (Customer Satisfaction Score), to name a few. See the list of UX metrics & KPIs to choose the most suitable ones for your case.

Trends and changes in data can tell you a lot about the current state of the app’s UX. In order to spot areas suffering from design debt, put this knowledge in the context of the historical background of your product:

• flows redesigned or designed by many teams over time
  
• places most influenced by the changes introduced to the business model
  

Visual design debt

The first method that usually comes up when teams try to quantify the visual inconsistencies across the product is to create a thorough interface inventory of each page or flow: calculate the percentage of visual misalignment going page-by-page or component-by-component. It can be useful but, in many cases, percentages may be hard to relate to, especially given that they are usually based on a rough estimate of page real estate and that visually inconsistent elements can be used across many pages.

Instead, it’s better to focus on the most prominent quantifiable symptom of the visual design debt: time spent on ensuring visual design quality.

How: measure additional time spent on design QA

Track time spent on:

• reviewing visual aspects of implementation during the design QA process (designers)
  
• fixing spotted UI inconsistencies (engineers)
  
• comparing and aligning existing styles with those provided in designs (engineers)
  

If your team doesn’t use time tracking software, you’ll need to ask designers and engineers to provide a perceived percentage to gather this data. It will not be as accurate but it will give you a fairly good understanding of the problem. If your organization uses a time tracking tool, you can add ‘visual QA’ to the project to distinguish it from other activities, especially the UX and code reviews.

To make it easier for your team to provide information, limit your research to 2–3 big projects only. Inform both designers and engineers about the need to track visual QA in advance.

Operational design debt

To measure operational design debt, you need to examine both internal and external processes that engage the design team.

• Internal processes
  
  ‍How: calculate the proportion between preparation and core work
  
  ‍Long ‘housekeeping’ before starting the design project indicates issues with resources and methods. When working with outdated files, incomplete component library and in general chaos, designers spend disproportionally a lot of time on:
  

• aligning styles, assets and components with the implementation
  
• creating mockups from scratch
  
• looking for and understanding past design files
  
• writing usability testing documentation from scratch,
  

instead of solving actual design problems.

To understand the scale of the issue, track time spent on preparation vs. core activities in each project.

As previously, you may need to ask your team to give the perceived percentage. When preparation exceeds 3%, it’s a red flag.

• External processes
  
  ‍How: measure time spent on solving preventable issues during the design handoff
  
  ‍Ask your team to track how long it takes to fix misunderstandings and do additional work caused by:
  

• not engaging developers in the earlier phases of the project
  
• discovering unknown constraints too late in the process
  
• missing details because of the vague DoD (definition of done) for the designs
  

If those problems delay your delivery by causing a significant difference between the estimated and actual time needed for the project, the operational design debt is probably a problem in your team.

Focus on design-debt may cause biases

• Bias towards taking action
  
  ‍Usually, once people understand what design debt is and its impact on products and organizations, they are naturally biased towards taking action. They want to pay off the design debt as quickly as possible to avoid the negative consequences. They forget that:
  

Reducing design debt does not guarantee the success of the product, it only betters the chances of avoiding failure.

Acknowledge what are your company goals to properly balance paying off design debt and working on other projects.

• Bias towards seeing design debt as a cause of every problem
  
  ‍Note that when doing research, your interpretation of facts may be skewed. You may discover a lot of additional issues and connect them incorrectly to design debt only because that’s your focus at the moment.
  

It’s important to distinguish badly designed experiences and processes from actual design debt.

Always look for the root cause of the problem. Dig deeper: cross-reference the issues, gain an understanding of changes introduced to the business and product over time, to find the actual cause.

Your work is never done

Understanding and measuring design debt is an ongoing process that gives the most value when continued over a long period of time. The initial examination shows the current state and gives a hint about what actions need to be taken to better the situation. It’s already a big contribution.

The positive effect on the product and organization grows exponentially when the work doesn’t stop. There is a chance of gaining a better understanding of the design debt’s impact on the product and the team. The strategy can be positively influenced by this knowledge and lead to better results overall.

Read the next article

Paying off Design Debt