Earlier this year we discussed metrics as a key component of getting live chat performance management right, geared towards an audience who may be unfamiliar with data collection and analysis. In this post, we’ll take a quick look at collecting baseline data.

If you are encountering challenges to high-quality data collection (especially as a result of poor data entry practices by your staff, and/or an inadequate CRM integration), please contact us.

As a reminder, data collection functions within the context of an active five-step process:

  1. deciding what to ask
  2. determining how to answer
  3. getting “Before” right (this is where data collection starts)
  4. reacting to “After”
  5. preparing for your next “After”

By the end of Step 2 , you’ve established the parameters of your problem and how you plan to approach solving it (including how you’ll measure your progress towards your solution).

To offer the clearest explanation, I’ll continue to ground this conversation in one of our examples from last time: reducing wait time and transfers via smarter routing. For this goal, we should track our success by monitoring wait times and/or transfers during the times of day or during the types of conversations that our team struggles with the most.

Step Three: “Before” (And Making Mistakes)

Gathering baseline data is the “Before” picture that you’ll compare to the “After” of your implemented solution. This comparison will help you to determine to what extent your solution achieved the desired result (e.g. reducing wait times and transfers). Because of this, it’s important that you understand the scope of your question (what to ask, e.g. can we reduce wait times/transfers?) and your solution (how to answer, e.g. adjust routing) prior to gathering baseline data (the “Before” picture).

If you didn’t establish the scope of your question and your solution before collecting your baseline data, you might not have thought to track transfers for specific topics or transfers during a specific time of day, vs. transfers as a general metric. This kind of oversight can result in less actionable data, as there may not be an obvious impact looking at overall trend data.

But what if the parameters you established as part of your solution’s scope require information that you don’t presently possess? For instance, perhaps you don’t currently track the reason for a transfer. This could be useful information if the goal is to reduce the number of transfers. You have two options. First, you could begin gathering that information now, and wait before actually implementing your proposed change until you’re comfortable with the baseline you’ve established. Alternatively, you can start collecting data at the same time that you implement your changes (“build the plane as you’re flying it”), and simply use the performance reporting (the “After”) from your first change as your baseline (“Before”) data to precede a second change.

If you’re just getting started with performance management, it’s probably going to be most productive for you to take the first option and start by simply collecting data, and getting comfortable with doing that in a new way. Once that data collection has become routine, then move on to the task of managing the new change process that will drive the continuous improvement of your wait times/transfers.

One final note about baseline data: in a pinch, you wouldn’t be the first organization to use industry benchmark data as your rough baseline, prior to actual data collection. If you’re on a tight timetable, you can probably get away with using industry standards as your baseline while you work to generate your own, organization-specific data.

Step 3.5 – More about Mistakes:

It’s entirely possible that your first attempt to gather baseline data will miss an important detail or extenuating circumstance.

Maybe many visitors get transferred because they selected the wrong department in a pre-chat survey. If you pay disproportionate attention to how your performance compares to a metric (e.g. number of transfers) rather than paying attention to the quality of metric itself (e.g. are you tracking reason for transfer?), you might continue to overlook an inaccuracy in your data that could easily be remedied (e.g. filtering out quick and harmless transfers resulting from user-error).

It’s much easier to correct this in the benchmark phase rather than retroactively during the performance management phase, because a change in the benchmark phase can cause a change in process that automatically fixes the problem going forward. A retroactive change during the performance management phase might require significant manual effort to correct reporting, or delay reporting until “cleaner” data has been collected for comparison.

Lingering Data Collection Questions

1. How much data do I need to correctly come to a conclusion?

The continuous improvement process is less about scientific rigor and more about making sure your team is paying attention to the right things and working consistently to improve them.

There may be seasonal differences or anomalous spikes or lulls in activity that could impact how long you should be collecting data before coming to a conclusion. For non-statisticians, a good general rule is to gather twice as much data as you plan to compare (e.g. if you want to see how things improved, or didn’t, between weeks two and three, look at data for weeks one through four). This is not proof of impact, but it is evidence that can guide the next few steps in the iterative process of performance management.

2. What are reasonable expectations for what the “Before”/”After” impact will look like?

This will vary immensely by the situation. Although unlikely, it’s certainly possible that you may have chosen to change something that won’t have any impact at all on the problem you’re trying to solve. It’s best to set the expectations for yourself and for others that you are committed to finding the solution, but that it may take a few tries to figure out what precisely that solution may look like, and then to optimize it.

3. Is the extra effort and infrastructure of data collection worth what it helps us to accomplish?

Yes and no. Yes, in the sense that it helps create a culture of continuous improvement within your organization. Also yes in the sense that you aren’t going to invest the time and effort into solving a problem that won’t have a tangible benefit. However, it is possible that you may spend so much time trying to crack the code on a business problem that the pain of your problem-solving process is worse than the pain of the initial problem itself. Be sure that the problem you are trying to solve is worth the time and trouble an iterative approach might ultimately represent!

4. How do I get buy-in from staff so that there aren’t data quality issues skewing my results?

This is the question of an experienced change-maker! You need to demonstrate that better data quality helps them directly, through making them better at their jobs, or ultimately making their jobs easier, or both. In our example, logging the reason for transfer ultimately resulting in having to do fewer transfers is an example of data quality making their jobs easier.


Next time, I’ll talk about the continuous improvement process of Steps Four and Five: adjusting what your team is doing in response to what you’re learning from the data you’ve been collecting, and then observing and adjusting again, and observing and adjusting again, etc.