Project / AI assisted user behavior analysis | Kai Brunner

The story
Designing an enterprise user experience for modeling and setting up automations created a curiosity in me for software taking on complex tasks where our human faculties limit output, or limit us in solving recurring problems altogether.

I wanted a starting notion of Artificial Intelligence and Machine Learning to continue thinking about the possibilities for automating design tasks and UX practices. Berkeley Haas offered executive classes for what I was looking for.

Overview
The following concept originated from experiencing the challenges of acquiring end-user feedback over the course of bringing an enterprise software product to maturity.

One of the most critical aspects of product management and design is getting timely and qualified feedback, when end users are several layers removed in the communication framework.

Under good circumstances, their feedback gets relayed through a variety of inquiry methods, sometimes more directly, through escalations to the support team. What if end user behaviors in the product could generate labeled data to be synthesized into actionable feedback.

Problem / Solution

Strategy & Business Value

Landscape
There are crowdsourcing user testing platforms that provide early feedback for features in development, or newly implemented features.

The value resides in accessing a pool of diverse testers that match the sought user criteria.

Limitations
These services work well for relatively simple consumer software products. Domain specific enterprise software however requires testers with domain expertise.

Such services do not provide feedback from ongoing usage across multiple tasks, and teams.

Opportunity
Data of individual end user and team collaboration usage are invaluable for product teams to make accurate evidence-based decisions towards evolving and improving their product.

Option #1 – Develop an in-house AI solution tailored to one or a suite of enterprise software products. This requires hiring and reorganizing leadership and management. (Smaller investment).

ROI – Use generated data for product roadmap. Sell reports to customers.

Option #2 – Build a company and business for an AI solution with plugins and APIs that allow to service the enterprise software market with a SaaS or PaaS offering. (Larger investment).

ROI – Growth relative to the number of subscribing customers.

Metrics

Approach Overview
The AI executes 2 actions. #1 it harvests data of individual user’s activity by recording cursor location and activities in the system. #2 it synthesizes both into a report of user behavior.

The analysis increases in value when combining the recorded data of multiple users to understand how they collaborate as a team.

The starting point is user intent. What task is a user wanting to accomplish and how successfully (in how much time) was it completed?

Successes and Failures
Based on the user’s intent for all manner of tasks, the measure = was the tasks successfully achieved or not. (This limits to actions within the system. It does not include the reading of dashboards or interpreting data visualizations).

Time spent on a task is assessed in context:

Login session
Start and end time of a task
Sequence of tasks
Interruptions (i.e. meetings; dependencies)

Technology

Data

Origination

The data required for the Machine Learning to parse and translate into actionable information is generated by end users of the enterprise software product and their activities within the system.

There is a threshold of activity that needs to be met by a minimum number of users with a level of proficiency in the software that encompasses all of its functionality. The threshold is met over time.

Ownership & Privacy

The data generated by system usage belongs to the customer.
With consent the data would be transferred to the software provider on conditions of anonymizing it .
The reports and analytics resulting from the AI synthesized data belong to the software provider.
The software provider has the right to combine the analytics of different customers for product development purposes.

Conceptually, the data would be labeled because it is generated within a closed and controlled system.

Human Factors

Organizational Change

The handling of data and compliance to privacy requires a dedicated leadership role and an IT team suited for data-driven product design processes.

Alignment in strategy and execution is necessary at the C suite level. Top management must have broad awareness of respective domain practices for effective collaborations.

Managers and their teams will be trained for interpreting data, and translating it into solutions that meet customers’ needs.