The Future of Market Research with ChatGPT: Thematic Coding

Posted by  Derek Jones

POSTED ON  August 10, 2023

In the fast-paced and dynamic market research environment, the ability to analyse data quickly and accurately is a precious skill. One emerging solution that we explored is the use of artificial intelligence, like ChatGPT, in coding open-ended responses. A journey with promise but also with challenges.

ChatGPT, an AI developed by OpenAI, promised a compelling solution to the often-laborious task of coding responses. Its core strength was in automation. Converting hours of manual coding into a swift, systematised procedure was a triumph of technology. Equally impressive was ChatGPT’s adaptability (albeit with lots of help).

The code frame, a structure used to categorise responses, was not static. Instead, we evolved it with the data, continually refined through human intervention for enhanced precision and relevancy. This dynamic adaptation of the code frame was crucial to navigating the richness and diversity of the responses.

Yet, like every new technological integration, our experiment with AI had its challenges. Data length limitations led us to process responses in batches. While this required additional planning and patience, it was crucial to managing the high volume of responses. ChatGPT sometimes diverged from the task, such as omitting data from tables or misjudging codes, but with close monitoring and immediate feedback, we could correct these diversions.

These challenges underscored an inherent limitation of AI: understanding the subtleties and context of human language and sentiment. In these instances, the irreplaceable value of human intervention became apparent. The coding needed regular human validation checks for accuracy, reinforcing the critical role of humans in the process.

Our AI-human partnership was highly productive and, to be honest, a lot of fun. We also leveraged ChatGPT’s capabilities for tasks like data formatting to turn unstructured data into easily readable comma-separated values (CSV), which could be readily integrated into software like Excel. This hybrid approach of using AI’s capabilities coupled with human validation checks made the coding process more efficient and precise.

So what worked?

  • Code frame building: ChatGPT’s ability to quickly analyse data and suggest initial themes considerably accelerated the process.
  • Automation: The automation of the coding process saved considerable time, enabling the focus to shift towards interpretation and decision making.
  • Adaptability: The code frame’s ability to dynamically evolve with the data and continually adapt to emerging themes was a major advantage. It ensured that the codes remained relevant and precise. This does, though, take human interaction.
  • Iterative improvement: The iterative process of coding, refining, and re-coding led to improvements in accuracy over time, demonstrating ChatGPT’s learning capability, although again requires a human to help with the process.

What didn’t work?

  • Inaccurate coding: There were instances where ChatGPT assigned incorrect codes to responses. This required manual checks to ensure accuracy.
  • Understanding nuances: ChatGPT sometimes missed the subtleties and nuances of human language and sentiment, leading to less than accurate interpretations.
  • Data volume: The inability to code large data sets in one go led to a need for batch processing.
  • Amnesia and embellishments: Sometimes, ChatGPT would lose track of the code frame or add extra details to the responses, necessitating human intervention.

Recommendations for improvement

  • Regular validation checks: Conducting regular validation checks can ensure accuracy and help in real-time course correction.
  • Training the model: ChatGPT benefits from training on a diverse range of open-ended responses to enhance its understanding of nuances and improve its coding accuracy.
  • Refinement of code frame: Continuous refinement of the code frame based on the feedback can help improve the accuracy of the codes.

What’s the takeaway from this exploration? AI tools like ChatGPT are robust assistants, capable of taking on routine tasks and streamlining the process. However, they’re not replacements for the human understanding needed in market research. The nuances of language and the context behind responses are still better comprehended by humans.

The beauty of using AI in market research lies in this symbiotic relationship. It’s not a solo act but rather a well-coordinated dance between AI’s speed and automation, and human intellect’s discerning understanding and intuition. Regular validation checks, continuous training, and consistent refinement of the code frame contribute to this balance, ensuring accuracy and efficiency.

In conclusion, the integration of AI into market research is an exciting prospect. Although it presents certain challenges, it also opens up new horizons, creating an enriched research process. This harmony between machine efficiency and human insight might well be the key to unlocking deeper understanding and greater opportunities in market research. And remember, just as a conductor guides an orchestra to a harmonious performance, human guidance is necessary to bring out the best from AI in market research.

But wait, there’s more! Taking everything we’ve learned so far, we’ve developed a 7-point plan for coding with ChatGPT. Download it by filling out the form below.

Share this post