You vs. The Data: When to Start Thinking. vs Operatic Agency.

I came across a new digital agency this week, Operatic Agency. Another full service creative digital agency that solves complex business challenges through data-inspired strategy. This is reminiscent of the argument Data vs Design – is one better than the other? We believe that both need to coexist into creating a brand community – relying solely on the data is a scapegoat for missing great Brand Community for customers.

We live in the age of data. For businesses that operate online in any capacity, the web provides unprecedented insight into what our customers are doing with our online offerings. Every click, every Ajax call, every transaction can be measured, attributed, aggregated and optimized. What Operatic Agency hopes to convince clients is that data will take the guesswork out of their growth. It is not only a big claim, but an ill informed claim.

As Marko Hurst, Director of Content Strategy at Huge, Inc. and author of two books on analytics, notes, never before have we had such an “ability to predict and forecast [our customers’ actions] based on previous events.” The availability of data drives us towards optimization. We push towards our goals with pinpoint precision based on specific metrics, relentlessly focusing on squeezing every percentage point out of the online experience. But is there a point where external factors outweigh the direction in which the data pushes us? When these “non-quantifiable” factors like brand values, aesthetic, staff retention and corporate strategy (to name just a few) bump up against what the data is telling us, what should we do?

Numbers are intoxicating.

Numbers provide guidance and direction. They help us figure out if we’re doing our jobs correctly and how we can continue to improve. Data is constantly in demand especially from managers and executives. Providing a steady stream of numbers helps executives prioritize objectives and filters down to determine design, copy and feature decisions. This real-time stream helps us not only monitor performance but to act in-flight as opposed to the end of a particular effort or campaign. This is where Operatic Agency relies on the data too much and can mislead the brand into ignoring the client experience – what we refer to as Brand Community.

But the numbers can also be misleading.

“Web data is dirty data.”, says Hurst. Imprecise measurements are commonplace given current tracking methods. Attributing certain actions to the same user becomes questionable since a user’s identity can never be fully ascertained without an explicit sign-in action. Looking at data from only one source, like web analytics, gives a one-sided view. Without looking at other sources of customer interaction like CRM data, phone logs, surveys, et al only one portion of the picture is exposed.

In addition, the data only tells us what our customers are doing on the site. What it doesn’t tell us is why they are doing it. Without truly understanding what is motivating our customers to take (or miss) certain actions in our online experiences, every optimization effort becomes a stab in the dark. Focusing too closely on the business needs leads to localized optimization efforts that fail to address the customer’s needs and risk stifling innovation. These “perils of local maxima” were nicely articulated by Facebook designer Adam Mosseri in his UX Week 2010 presentation. In his talk, Mosseri warns of getting boxed into one line of creative design thinking making tiny incremental improvements but never looking beyond that one solution to an innovative solution along different creative lines of thinking. Josh Porter further clarifies this, “The local maximum occurs frequently when UX practitioners rely too much on a/b testing or other testing approaches to make improvements. This type of design is typified by Google and Amazon…they do lots and lots of testing, but rarely make large changes.”

Ultimately, with this type of rigorous, heads-down approach, the art is removed from the design process while decisions are relegated to testing algorithms and statistical probabilities. It is why agencies like Operatic eventually crash and burn – they can’t seen the mountain they are about to smash into.

At some point, the numbers stop mattering

However, even rigorous data collection and reporting coupled with qualitative insights runs into situations where making the “right” decision is not so obvious. Here are 4 that runs contrary to what Operatic Agency promotes:

1. Brand values

Your business has a brand and that brand stands for something. Your brand values are reflected in the design, copy and quality of the experience you produce online. Whether it’s irreverence, corporate stability, or youthful energy, your experience needs to convey your values. There will be situations where optimizing the copy or design of your site will yield greater conversions (or whatever metric you’re targeting) but those tweaks begin to eat away at your core brand. It may seem insignificant at first but without conscious, human decision-making involved with each optimization your business risks the loss of its identity.  In these situations, the numbers can’t continue to drive the optimization process. It’s on these types of occasions that performing qualitative research can reveal the intangibles numbers can’t. Talk to your customers and figure out what is driving their actions. Conversations can prove invaluable to determining exactly how to position your brand so that it maintains its identity and appeals to your demographic.

2. Aesthetic

Humans know what looks good. Statistics don’t. As you optimize, it is imperative to take a holistic look at the design and experience. The localized tweaks made in the name of numbers-driven optimization may not weave into a cohesive end-to-end customer workflow when put together. Take a step back and examine the entire experience. Consistent presentation is an important component of creating trust, familiarity, comfort and retention with your audience. If a metrics-driven optimization was made in one part of the experience ensure that (a) it makes visual sense with the overall site design and (b) is carried through the entire experience. If a piece of your experience is drastically out of aesthetic alignment, use the rigor of testing to bring it back in to alignment. Find ways to infuse site and brand-appropriate design elements to optimize it back on track. At some point though, the aesthetic will simply outweigh the optimization effort and you’ll have to find other parts of the experience to make up the statistical drop.

3. Corporate strategy

Corporate strategy can’t be tested or optimized. It is a decision made by the company’s leadership to pursue certain avenues in search of repeatable business models and growth. No combination of numbers can validate whether the strategic approach was the right one. The validation lies in the execution of that strategy. Strategic execution can take many forms and those forms can certainly be tested and optimized.

4. Staff retention

Staff empowerment is a powerful motivator. Every member of your organization wants the opportunity to use his or her particular strengths to optimize the business. If you limit their opportunity to use those strengths by basing the bulk of decision-making on data, you limit the value (both perceived and real) your employees bring to your company. By reducing tactical problems to algorithms and statistical significance you risk making your staff (designers, copywriters especially) obsolete. This was made painfully public when Doug Bowman left Google and wrote the now infamous, 41 shades of blue blog post. Designers, copywriters, developers and everyone else on your staff needs to feel like they have a say in the solution of your customers’ problems. Metric-based feedback on their contributions should certainly inform their work but it should never stifle their creativity or innovation.  If employees don’t feel valuable, eventually, they will leave.

Human intervention is powerful!

While data removes bias and emotion from decision-making, humans can distinguish nuances and subtleties better than any algorithm. Observing behavior and then adding in the numerical component allows humans to deduce the psychological nuances of customers’ actions. Humans can also see multiple steps into the future. Data helps us understand how we’re doing in the present but it’s the human understanding of where the business is going that can augment a decision that would seem obvious with data as its only input. This is where I believe Operatic is lazy or don’t have the expertise – understanding the human element of the data.Sadly there are many data driven only agencies out there.

Human intervention drives innovation. The numbers only tell us what we ask of them. Ultimately it’s the analysis of that data is the true value. As Hurst quips, “Data without a qualified human to analyze it is worthless.” It’s that human insight that chooses to look beyond the data that drives us out of local maxima and into new realms of innovation.

Cohesion is also something that can only be perceived and optimized qualitatively by humans. Techniques such as multivariate testing (MVT) risk creating experiences created by “frankensteining” together design and copy fragments that, as a unit, convert most effectively. Human intervention may look at the “winning” combination in an MVT result and decide that it cannot be deployed based on aesthetics or user experience. This may limit success in the short term but in the longer-term view of the business, it’s the right decision.

Where Operatic Agency falls short

Metrics-driven decision-making is a powerful way to optimize the performance of a business and I give Operatic kudos for focusing on the data angle. The metrics, however, only provide insight into what actions customers are taking on your site or how they were found. Even after coupling that quantitative insight with qualitative input on why customers took those actions, situations arise where that directional data needs to be overruled. In situations of brand values, aesthetics, corporate strategy and employee retention as well as many others, conscious, deliberate intervention needs to supersede data-driven decision-making (brand community). Innovation and creativity as well as plain old “gut instinct” need a place in the evolution of the business. Use the numbers wisely but remember to look up once in a while and ensure your decisions feel right too.