I don't think I'm trying to level up my teamwork (at least that's not language that resonates with me), I'm just trying to stay in close, less-formal than an email, communication with my coworkers!

Carvana: recently I had a car to sell and I was getting low-ball offers from the dealers. I'd seen Carvana trucks driving on the highway with messages telling me it was easy and they were straight shooters (aka trustworthy). I went to their site, found the 3rd CTA on the page was for me. 2 minutes and they'll come pick it up, I don't have to spend time on the phone and worry about taking it somewhere? The beginning (get the offer) and the end (handing off the car) were the most stress inducing parts for me, and they were promising a low-stress option (the offer was great, by the way, and they delivered on the stress-free sale all the way. Would recommend 5 out of 5!)

Boardable: I am always happy to lay a critical eye on my own company's product and chide my coworkers, so let's add it to the mix here. This is the product that I need to measure the "promise" and onboarding success of users and customers, so knowing the ins and outs of how the product is positioned and how the team thinks about its value propositions is foundational to this endeavor.

Boardable is a Board Management portal, originally positioned as a "Board Engagement" solution. "From Friction to Focus" appears to be the primary promise, and internally we often discuss it as reducing the chaos of managing many people that are often geographically dispersed, where the primary competition is "digital duct-tape" (aka a mix of Dropbox, Google Drive, Zoom, email, Survey Monkey, and personal calendars). In this case we can't measure it as X messages sent, or 1 car sold, because "reducing friction" or "obtaining focus" is nebulous and probably means many things to many different people.

How do we measure it?

Retained: to complete this analysis we need a working definition of what qualifies an active, successfully onboarded user, and the definition can be refined later. We need enough data points (enough users) to assert confidence in the results, so one approach could be something like "the most active Y% of users that gives us at least N users." Measuring "active" as logins per month, or some other generic metric, is sufficient. Another, which I use, is "all active users with a 6+ month tenure", where 'active' is more than 1 log in every month.

If you can identify a set of retained customers we can theoretically work backwards to finding the "onboarded to the promise" metrics. What did these retained (presumably happy with the product) users do in their first 90 days? 

But where does onboarding start, one might ask? Don't over think it! I'd start from the first time they created an account or installed and opened the app. If you have enough samples (accounts) then idiosyncratic behaviors and gaps between getting started and onboarding all the way will wash out over the whole data set. Remember, we're comparing across a breadth of long-term, retained users who we can hypothesize have successfully found the promise in the product (as they understand it) or else why are they still actively using it?

When looking at a particular action (sending a message, scheduling a meeting, uploading a document, etc.) active users did during their first 90 days we typically find a distribution with a long one-sided tail. Most of these happy, retained users found the promise with a smaller amount of activity, a few really got into it and did our thing lots and lots of times, and some probably didn't do this particular action

One of the primary features of the Boardable platform is cloud-based document storage, helping users consolidate their board related materials ("from friction to focus"). In the first 90 days (see histogram graph) lots of active users uploaded fewer than 100 documents but a smaller number of users went on to multiple hundreds. Because of this long tail (the highly active users) an average will skew to the high side (average=71). 

As a number to assess the onboarding achievement of new customers a goal post of 70 is unreasonable, requiring the new user to upload 3 documents every 4 days. We know from customer and prospect interviews that users upload documents at two times: a tranche of organizational policy type documents when they first start, and then a set of reports and materials around each monthly/quarterly/yearly board meeting. Clearly some customers were very active in their first three months, but many others had far fewer documents to store and share. To obtain a more realistic goal post for onboarding assessment we could trim the long tail first, based on some percentage, or take the median of the set. In this case median=32.

How Fast?

What is an ideal speed to get a customer to this onboarding success mark of 32 documents? How quickly are they getting to this "promised land" of product value? We've looked across all long-term active, healthy customers and found that the median account uploads 32 documents in their first 90 days. However, some will hit 32 in their first week, and some will take longer than 90 days.

We con do the same thing, counting the days between all of these user's first document and their 32nd document. It has a very similar distribution with some users taking 90+ days (forming another long tail) and some getting there in a couple days. Taking the median gets us: 60 days!

Conclusion

From this analysis I can report that a good sign of long-term health for a Boardable customer account is uploading around 30 documents within their first 3 months. We can understand this 'story,' if you will, as a proof point that the new customer is "reducing friction" (a promise) and finding value in the platform by storing and sharing their materials and documents. Certainly if a user has not uploaded any documents in their first 90 days they have not found value there, and if they exceed the 30 document mark then we can deprioritize outreach to them and focus on the ones that may need more resources, training, or support.

Secondly, while 30 in 3 months is the rule of thumb, the median long-term successful customer actually gets there in 2 months. That would be a helpful expectation to set both with the customer and internal teams to proactively work towards more accounts reaching the land of product promise!

Finally, this analysis should be repeated for every potential primary activity that ties to the promise and product value propositions. At Boardable this includes: meetings scheduled and held, agendas assembled and published, discussions started and replied to, and more. Each of these primary actions is charted out with onboarding milestones at 30, 60, and 90 days, which inform a very robust process to ensure as many customers as possible realize the promise and make the progress they are hiring Boardable for.

Dr. Ben Smith is a Data Scientist and thinker, fascinated by the appearance of computers in our daily lives, creativity, and human struggles. He has had the privilege to think, learn, and write at the University of Illinois, the National Center for Supercomputing Applications, the Cleveland Institute of Art, Case Western Reserve U., IUPUI, and at Boardable: Board Management Software, Inc.

If you have feedback or questions please use Contact me to get in touch. I welcome thoughtful responses and constructive critique.

This will pull up every click or tap on anything on the specified domain (boardable.com), while matching the path "/academy" to isolate for a single page. As of February 2022 this page's design looks like this:

Going to Heap's app again, we can setup a report that gives us all clicks on this page over the last 30 days, broken out by Target Text (what was in the button, link, or text that the person interacted with). This is the Report definition:

On my site this produces a list like the one on the right (from most frequent to least, the top 9 results shown here).

Clicking the Right Things

The first several can be seen as the primary CTAs on the page ("Board Member Academy"), indicating interaction with elements that were intended by the designers ("Meetings," "Rollout," and "Advanced" are below the fold). However, what is the long string starting with "Boardable Basics Quick-start ..."?

It turns out a significant number of site visitors click on the header text and sub-title on the page, which are not interactive elements. It is just text and does not take the browser anywhere. However, you can quickly see why: the text style is different (especially the lighter blue subtitle), it is prominently placed, and could easily be understood as a link to more content. 'Yes! I came here for the basics and I want a quick start, let's go!' This is definitely an opportunity to get visitors to something about "Boardable Basics" or a "Quick-start" guide, since visitors are already clicking on it.

As you might anticipate next, the "Video Tutorials" inverted white text header also receives a lot of clicks but is not actually a link.

Clicking None

What about non-textual interactive elements? Or visitors clicking on non-interactive, non-textual elements?

Indeed, in this report the top grouping has a Target Text value of None, accounting for 12.5% of all clicks (over a 30 day period). Heap will also provide Target ID, Tag, and Class, and in this case Tag gives the most complete view:

One third are clicking on <button> tags (good!), one third are clicking on <img> tags (many of which may not be links), and the rest are on other graphical or input elements. Considering the large number of image-based buttons on the page, we can hypothesize that visitors are engaging with those as designed.

The primary opportunity for improvement indicated here would be to ensure that every image <img> on the page is also a link (taking the visitor to the same content as the nearest CTA). For example, the large "Boardable Academy" mortarboard hat logo is non-interactive and does nothing, but it could link to the same page as the smaller Board Member Academy button next to it.

Empty Folders

A very similar discovery showed up when following the same process with Boardable's web application (as of writing anyone with a valid email address can signup for a free trial here. Go check it out.)

Using a "Click Any" event in the app and then breaking them out by Target Text again provided some anomalous results. Fortunately, most of the list of top-50 clicked elements across the product fit with the prominent interaction points as designed by the developers. However, here is an example of an unexpected bread-crumb that lead to product insight and usability improvements.

In the list of document related content one thing people click on is not a button:

• documents/folder: Save      

• documents/folder: Select File  

• documents/folder: Submit  

• documents/folder: This folder is empty  

• documents/folder: Upload File

Four of these are clearly actions a user can take, but "This folder is empty" is not an action, it is a declarative statement. Where is it, and why or what are users expecting it to accomplish?

As of writing (February 2022), the page in question is the view of a new folder in Boardable's Document portal: 

It was a matter of a few minutes comparison with similar tools (Dropbox, Google Drive, etc.) to reveal that "this folder is empty" is commonly the drag-and-drop upload interaction element in other products. We can hypothesize that users familiar with these other tools are expecting to initiate an upload process and proceeding to select a file by clicking on the message (and, again, the visually distinct nature of that text on the page is could be a cue to interactivity, which would be misleading).

In both of these cases we were able to find these opportunities for improvement by looking at what users actually clicked on, rather than assuming users are clicking on what we want them to. I can set up funnel conversion metrics on uploading documents, or consuming "Quick-start" material, but if users are getting stuck trying to find the interactive elements I am already starting at a disadvantage.

Am I measuring the right things? I think I am 2 steps closer. And I am going to keep on searching, digging, and questioning the assumptions I can uncover, that is for sure.

Dr. Ben Smith is a Data Scientist and thinker, fascinated by the appearance of computers in our daily lives, creativity, and human struggles. He has had the privilege to think, learn, and write at the University of Illinois, the National Center for Supercomputing Applications, the Cleveland Institute of Art, Case Western Reserve U., IUPUI, and at Boardable: Board Management Software, Inc.

If you have feedback or questions please use Contact me to get in touch. I welcome thoughtful responses and constructive critique.

With a simple cleanup.py in a /common directory. At first it was empty, but has become a powerful aid as the EL landscape became more complex. Cleaning up temporary files, closing any SQL or SSH connections, etc. can all be taken care of in one place.

created_at & updated_at

This is more of a database management idea, but including timestamp columns for each entry created_at and updated_at times should be the default for every table. I have it written into the load modules to add those columns on any DataFrames headed for the warehouse, and check to update the updated_at entries on any UPDATE SQL calls.

In addition to querying for the most recent data for analysis, it is also incredibly helpful in identifying stale data or cases where jobs failed silently. If we're expecting every table to have at least some updated records every day it's a quick matter to check for tables with old maximum updated_at dates and locate the failing job.

Soft Delete

Another database idea: in many cases hard deleting data is necessary, to keep only the up-to-date, current state of data. However, soft-deletion should be kept in mind whenever possible. This is a process of keeping a "deleted_at" column in the table and performing something like:

HubSpot (a CRM I've worked with extensively) includes just such an indicator with most of the data object types that can be fetched via their API. However, the bulk endpoints only return non-deleted records, so for a while I was confused why deleting things in the web app did not change the corresponding records in our warehouse. It turns out one has to request EACH record individually to get the deleted status. 🤦🏼‍♂️

Logging

Look at configuring Logging to write to a directory. I use a TimedRotatingFileHandler configured instance to make the files manageable over months.

Log every call, every request, every result. I have found this immensely valuable in both assessing the inefficiencies of my data scripts, but also in finding breaks and showing my work to others across the business.

Given a single interface method for the Extract stage, it's robust to place a decorator function around it to capture:

• Calling function name and module name

• Length of arguments (will catch length of query or curl request)

• Time of initiation

• Execution time

• Returned data size

This can be cached and pushed to a warehouse logging table on job cleanup!

The code I have in production is patterned like this:

Never Fail

The EL job itself cannot error out part way through. Jobs failing makes the possibility for gaps, duplicates, and erroneous entries shoot up dramatically. It makes a mess and is hard to clean up. Obviously careful but frequent use of try/except/finally blocks is critical, and getting the nesting right is important (do we try-wrap each individual call or the request pagination loop as a whole? what do we do with partial results? Etc.) Response codes 400+ won't necessarily break the script, but need to be handled and logged too.

When something fails we want to note what failed (the query or request that failed should be logged in its entirety), what the error messages were (lots of logging.error()), and what we're doing about it (throwing away the partial results? Trying again? Moving on to the next job?).

In order to never fail I've become increasingly fond of making any non-numeric data into text (VARCHAR, in the postgres world). This requires a lot of casting to use the data, but it ensures the EL jobs perform with a much higher reliability. I've lost far too much time cleaning up after someone has managed to put text into what should have been a number or date property, causing the database to reject the entire Load. Instead of relying on the system (CRM, billing, BI, whatever) to enforce data types, we just absorb it all, complete the Load, and worry about checking for empty or wrong-type data when we're analyzing or building reports.

At some point I'll have an entire post about the number of different ways one can cast a text entry to a timestamp in SQL. Who knew there were so many ways to skin the proverbial cat?

Reporting

Once everything is logged I think about ETL reporting in two ways, for me and for everyone else.

For me: I look at daily and weekly views of how much data is going where, which calls are most frequent, which sources take the longest, on and on. How are the ETL jobs performing? Which ones are the most efficient and which ones could use better optimization? Also, if something breaks I turn to my ETL warehouse logs to find the source, the scope, and the fix.

For others: ideally the data is shipped around behind the scenes, under the hood, and no one else needs to be aware and fully cognizant of how it all works. However, it takes a significant amount of work to get the engine humming smoothly and showing this work as valuable is necessary to getting support and recognition of the job. It may not be especially tangible, but reporting high level metrics of how much data is shipped each night, how many database calls, etc. gives some notion of how many gears and motors and drive belts are spinning away keeping data up to date and centralized.

De-duplication

Often we have data with a unique id key for the table (such as Contact ID, Account ID, event ID) and if the record changes we want to update the entry and keep a single row for each ID value. Some databases will help enforce this, but AWS Redshift does not. Here's the bit of SQL I use to perform a de-duplication job on a table (note: relies on accurate "updated_at" timestamps to identify the most recent of each duplicate entry):

We're creating a temporary table containing any duplicate entry IDs and their updated_at timestamps. (Replace the handle bar {{}} variables with your actual table and id names.)

Finally, we use the temporary table to delete the old, duplicate entries from the original table:

Dr. Ben Smith is a Data Scientist and thinker, fascinated by the appearance of computers in our daily lives, creativity, and human struggles. He has had the privilege to think, learn, and write at the University of Illinois, the National Center for Supercomputing Applications, the Cleveland Institute of Art, Case Western Reserve U., IUPUI, and at Boardable: Board Management Software, Inc.

If you have feedback or questions please use Contact me to get in touch. I welcome thoughtful responses and constructive critique.

Bubbles: what to do with size? In this case I decided to use the size of each action-bubble to indicate confidence (based on data volume). Actions that are almost never done and we have very little data on are small, while actions that are done by nearly every account are the largest. As new features are released we can see in the bubble size how reliable they are, and move the discussion towards decision making and away from interrogating the model.

Understanding Accounts

How do accounts gain their score? 

Rolling up lots of actions to a single score can enable automations and sales efficiencies, but not all scores are created equal. Is an account doing a low-score action many times, or a high-score action once? To catch this notion broadly, I broke the actions down into 5 color-bands according to their predictive value (black=required/necessary, yellow=predictive/sufficient) for plotting purposes, and then charted the incremental score amounts for each active account. The resulting bars (one for each account) provide an at-a-glance look at what accounts are doing, in terms of "activation." (in this case I found it interesting that a few lower scored accounts had significant amounts of orange, while a few higher scored accounts had very little orange/yellow.)

While there are clear patterns in the ratios between color sections in each account, there are a few interesting variants where some lower scored accounts have disproportionate amounts of orange, and a few higher scored accounts have very little orange/yellow. This leads to our next opportunity for analysis: what are those out of sequence accounts (with more orange/purple than the neighboring bars) doing in the product?

My model has grown to include over 500 observable product actions (from logging in to starting a Support chat to joining a video call), and attempting to read an event log transcript for an account is too opaque to quickly assess what an account is doing. Using the PQL scoring model we can break down each active account and each element that gives them some amount of score, and sum these up according to product-area (as determined by our team: Home, Documents, etc.) In addition to a dashboard to plot these amounts for each account, this now allows us to translate the visual into a quick textual description of the score.

Signals: once we can see the primary areas accounts are activating in, we can setup a little algorithm to write out a human-readable summary of what each account did (this idea was inspired by the great products from MadKudu.com. I created an action statement for each product area, then took the most important lines to create a text snippet that could be seen at a glance on each account's CRM record. For example:

Added 4 new users.
1 New user logged in.
Visited for 8 sessions.
Scheduled 3 Meetings.
Uploaded 4 Documents.
Created 1 Agenda.

Understanding Product

How is each area of the product performing in producing PQL value? 

In addition to slicing active account scores by PQL elements, we can also look at the product overall and ask: which areas of the product are most effective in "activating" users? I have found that some areas of the product at Boardable, while showing as highly valuable in predicting conversion (high sufficiency, low necessity) are done very infrequently and very few accounts get that far into the product. At the same time, across the breadth of less predictive elements some are performed far more often than others. By breaking down any set of scores (I like to take 30 day cohorts at a time) in total by product area, we can see how the parts of the product are performing in delivering Product-Qualification value. I find this especially useful to inform new product development discussions, as well as providing insight into the impact of new releases and feature updates.

Impacts

Sales: given the underlying premise of predicting conversion the PQL score is clearly indicated as a tool for Sales. It has proven powerful at Boardable both in the hands of people working to close deals, and to drive behavioral-based automation. Accounts that are not showing "activation" can be sent more general nurture material, while highly engaged accounts are better targets for specific product messaging. By breaking down what individual accounts have done, product "walkthroughs" can be crafted and delivered to steer these activated users into finding increased value.

Support: as a tool to give a broad-strokes picture of how engaged an account is, the PQL score can be useful to include on account records used by support personnel. Is this person asking this question with a very low PQL score, indicating they are a very light user? Or is this highly scored person asking with a lot of product knowledge, because the 30th document they uploaded hit a bug? This little bit of insight can give your team more of the user's background when picking up a support ticket (see: PQL Signals, above).

Going past conversion: while the model is built on pre-conversion correlations, it continues to deliver insight into account health post-conversion. Every user is on an onboarding journey with your product, and some may develop extensive usage knowledge prior to buying while others buy in anticipation of learning to use it. In B2B SaaS often a couple users will make the acquisition ("hire" the product) but then have a cohort of teammates that onboard later, at their own individual speeds. The areas of the product identified by our PQL model at Boardable align very closely with our onboarding milestones and "account health" scores, post-conversion. For example: uploading their first document is a key indicator of Product-Qualification and a significant initial milestone on the onboarding path.

PQL scoring has proven to be a great tool, but it also has many limitations and there are many things it cannot tell us (or leads to assumptions that may not be true). In Part 3 (tbd) I'll share my thoughts and experiences on the dangers of trying to use this particular hammer on things that turn out not to be nails!

Dr. Ben Smith is a Data Scientist and thinker, fascinated by the appearance of computers in our daily lives, creativity, and human struggles. He has had the privilege to think, learn, and write at the University of Illinois, the National Center for Supercomputing Applications, the Cleveland Institute of Art, Case Western Reserve U., IUPUI, and at Boardable: Board Management Software, Inc.

If you have feedback or questions please use Contact me to get in touch. I welcome thoughtful responses and constructive critique.

Conversion to a paid account is not important in trying to assess the health of a user base, from a usage perspective. In the user's experience having to go from a free trial or freemium to paid is friction, and if they have not formed any habits purchasing is speculative. If they do not quickly realize the value your product provides them they will churn. See my predictive Product-Qualified Lead scoring model for a conversion oriented product analysis.

Here I want to focus on retention, and go beyond Monthly, Weekly, Daily Active Users (MAU, WAU, DAU) and the common retention curve graphs. These are common SaaS performance metrics, but in order to meaningfully drive growth we need to dig deep and figure out what users are coming back to do over and over (the "use" aspect of "user"!) We are not random in how we use products, and after an exploratory phase (when we figure out what a product does) we use them to do things (where "doing" is understood to have a functional as well as emotional and social components). This post will start with assumption that we have users of a product already and we want to understand their habits, and what they actually do when they pick up our app every day or week (then as a next step we could use that knowledge to encourage new users in similar directions, or focus the efforts of our product and development team). How habits are formed is a fascinating topic as well (see this post for a good discussion of habit formation around software products).

Defining Habits

From Oxford Languages:

Habit: a settled or regular tendency or practice, especially one that is hard to give up.

From Merriam-Webster:

Habit: a usual way of behaving: something that a person does often in a regular and repeated way.

From Wikipedia:

Habit: is a routine of behavior that is repeated regularly and tends to occur subconsciously.

A habit, if viewed externally, appears as a reoccurring pattern, often with some notion of "automatic" and without explicit intention. Pattern itself is a reoccurrence of consistent elements over a space or time (Wikipedia: "A pattern is a regularity in the world...the elements of a pattern repeat in a predictable manner"). If we think about habits in our daily lives they can be described, typically, as a sequence of events that define this pattern:

Example: I drink 2 cups of hot tea every morning. I go to the kitchen, I pick up the kettle, I walk to the sink, I fill up the kettle (often, even if it already has enough water), I take it to the stove, put it on the front-left burner, turn the element on to max, walk to the shelf and take my favorite ceramic mug, put the mug on the counter, open the jar of tea bags, take one bag, put it in the mug, wait for the water to boil, and on and on.

I do this same sequence of actions almost subconsciously every morning. Someone viewing me everyday would be able to ascertain the "pattern" after they saw it repeated several times over several days. We could then say: Ben is a retained user (he is DAU of tea kettles), and has a strong habit-loop around tea making. [Interestingly, there is a lot of debate around whether we can self-report habits, since they are often stimuli based responses performed unintentionally. So this external observer is necessary, and in this case my family confirms my tea drinking habit. 🫖]

Sequential dependency-the precise order of actions in a habit is often not as significant as the co-occurrence of the actions. I can put the tea bag in my mug before I turn the kettle on, or after I've already poured the boiling water in the mug. The habitual pattern involves both actions but they can happen in a variety of orders. The common approach to catching this in Machine Learning is to count the number of occurrences of an action (also called a "feature") within a "sample" of the pattern. We can then compare distances between vectors of features to determine similarity and look for clusters of patterning.

Habits in software usage follow the same patterning paradigm, and with the right processing can be discovered and used for a variety of business decisions and tasks (lead qualification, opportunity identification, onboarding performance, account health and churn risk, etc.) For example: once we have analyzed for and identified a few primary habit patterns product users have formed, we can compare new users and watch as their usage settles into the same patterns. If their usage remains chaotic (incoherent, in a sense) then they aren't likely to be interested in purchasing. If they match the patterns of long-term users we can start making assumptions about their likelihood to cancel or churn (from a usage perspective). Additionally, we can test hypotheses on where the value in the product really is, and what keeps people coming back to use it again and again.

Example: I am a DAU of Spotify for listening to music. I open the app on my phone, I tap the Home button, and look at my 6 most recently played playlists. If one of those seems interesting I tap it, select Play or Shuffle-play, wait until I hear music, and turn my phone off. If I am looking for something new I start scrolling down, viewing all the recommendations, or use the search function until I arrive at a suitable selection. There are many other features and actions I can take in the app, but I only use a handful and a typical "session" for me involves 2-6 taps or clicks before I have obtained the value I am seeking: music to accompany working or cooking or driving.

In part 2 (TBD!) I will describe how to pre-process user data and perform a pattern analysis, identifying "habit signals" that can give us deeper insights into how people use a product and which feature combinations are most critical in delivering value. I will get into figuring out how users are using the product, which features are really valuable, and what habits we can see them building around the product that make it sticky.

Dr. Ben Smith is a Data Scientist and thinker, fascinated by the appearance of computers in our daily lives, creativity, and human struggles. He has had the privilege to think, learn, and write at the University of Illinois, the National Center for Supercomputing Applications, the Cleveland Institute of Art, Case Western Reserve U., IUPUI, and at Boardable: Board Management Software, Inc.

If you have feedback or questions please use Contact me to get in touch. I welcome thoughtful responses and constructive critique.

• See the end of this post for how to implement this on a database containing data sync'd from Heap. If you have data from another system the SQL should be reasonably modifiable to accomplish the same thing.

Theory

What we are looking for are actions in the product that indicate adoption, that the user is an "activated" user and is likely to upgrade to a paid account (or if we have packaged it well then they will see the value in the paid access). The nuts-and-bolts actions that we can auto-capture roll up to activities that make the product useful, like sending messages, adding friends, or putting items in a cart. Once we have the event tables filled out (see above), we leverage the logical relationships of necessity and sufficiency to form the building blocks for this model.

Necessity: a necessary condition is one that must true in order for another to also be true. When talking about product actions this can be understood as required events. If the only way to click the 'buy' button is to sign up for an account then the 'sign up' action is necessary for 'buy.' Everyone who clicks 'buy' will have already completed 'sign up.'

Sufficient: a sufficient condition implies another, but is not required. At Boardable 'delete document' is a sufficient condition for 'upload document' (you can't delete something that is not there), but if someone hasn't deleted anything they may or may not have already uploaded something.

A fully necessary action is one that all converted users completed. It is a required action, like signing up or entering a payment method, indicating progress towards conversion but with little predictive value. In set terms: all converted users are a subset of all users who completed the action. 

Rather than simply look at "how many people do the action and convert?", we instead look at the conversion population in the field of users that either did the action at least once, or never did it (see next image, the bounds of the paper represent all users in our specific time period). This catches people who maybe never saw the feature in the product and were unaware it was a capability. 

How the converted circle sits across the did-the-action/did-not-do-the-action field gives us a picture of necessity. To the degree that the conversion bubble sits in the "action" side it is necessary, and if the bubble sits in the "no action" side it is completely unnecessary.

Necessity

To calculate necessity we look at how many accounts converted without doing the action. If no one converts without doing the action we can assert that it is a necessary step on the path to converting (such as selecting a payment method).

At the limits, N = 0 when no one has converted and skipped this action (making it fully necessary), and N = 1 when everyone who has skipped this action converted (making it unnecessary).

Sufficiency

To obtain sufficiency we look at everyone who did the action and ask "how many accounts that did the action converted?" This can be plotted as a ratio of "Did Action" samples in the field of converted vs. dropped off users. If this ratio is higher than the baseline overall conversion rate then we have a valuable action!

To the degree that the action bubble sits in the converted side it is sufficient and we could say "doing action X implies conversion," but not "everyone who converted did action X." Once we know how many of the "did the action" users converted vs failed we have a useful calculation of "sufficiency":

S = Ac / (Ac + Nc) is incorrect, stated as "out of all the users that converted, how many did this action?" This is a Necessity calculation, again.

In this formulation, S = 0 (Ac ~ 0) only when no one does the action and converts (aka it's a conversion killer action, the "cancel now" button), and S = 1 (Nc ~ 0) when it is a necessary action (everyone that converts does the action), which we already capture above.

At the limits, S = 0 when everyone who does the action drops off (Ac approaches 0), and S = 1 when everyone who does the action converts (Ad approaches 0).

We can now define 3 categories of actions:

• Necessary but not Sufficient (N ~ 0, S ~ 0): required actions, indicating average conversion if a user does the action, and very low chance if they don't do it (account creation, selecting payment, etc.)

• Necessary and Sufficient (0 < N < 1, 0 < S < 1): actions that indicate engagement. Low chance of conversion if the user hasn't done it, and a high chance if they do.

• Sufficient but not Necessary (N ~ 1, 0 < S < 1): more users that do these actions end up converting than not, but they are not required. These are the most interesting.

Application

I have had great success splitting actions into these 3 categories (along the lines of the 'Activation, Engagement, Acceleration' framework described by Madkudu), and then creating a "score" for each active account. I worked out a formula that combines both the N & S calculations to create a single estimate of the Product-Qualifying value for each action, and then sum these values to create an account's Product-Qualified Lead score.

Finally, all active accounts are broken down into Low, Medium, and High PQL groups in order to deliver targeted messaging and automation routing. Continue on to Part 2 of this post series for more on delivering value across teams using PQL scores.

Code

The following breaks down the calculations in SQL sub-queries (this is Redshift SQL, a variant of PostgreSQL). The data it operates on consists of:

• ."Users" table with a unique entry for each individual user, containing

  • "user_id": a unique identifier

  • "trial_ends_at": a custom property that indicates the date the user's free trial ended. We only want events that happened during each user's free trial, not things they do after converting.

  • "plan": a custom property that indicates the user converted to a paid account. Change this to whatever you have that indicates conversion.

• An event table, "sign_up_complete" in this example. The event table has 2 data points we need: 

  • "user_id" that matches the unique identifiers on the users table

  • "time" when the event took place

Dr. Ben Smith is a Data Scientist and thinker, fascinated by the appearance of computers in our daily lives, creativity, and human struggles. He has had the privilege to think, learn, and write at the University of Illinois, the National Center for Supercomputing Applications, the Cleveland Institute of Art, Case Western Reserve U., IUPUI, and at Boardable: Board Management Software, Inc.

If you have feedback or questions please use Contact me to get in touch. I welcome thoughtful responses and constructive critique.

Playing Piano

As we received our first assignment to create a piece of music with the synths, our professor asked us to split into three groups:

1. All the skilled pianists; if you could play anything by a classical master from memory you were here. This was a small group.

2. Anyone who had taken piano lessons, or "knew enough to be dangerous." I was in this group!

3. Piano, what? If you didn't know the difference between black and white keys this was you. I was so excited not to be in the newbie group.

The professor proceeded to dismiss the first group: "I'm not worried about you. You know how keyboards are used in western music and you have the technical skill to execute whatever you want to play."

Then he dismissed the third group: "I'm not worried about you either. These keys? They're just buttons. Push them. Don't worry about how pianos are played, instead explore the sounds you get from your own individual approach. There's no wrong way to make music here."

Finally, he kept the second group over: "You are going to be the most challenged. You see a piano, when in fact they're just buttons. You've been told how to play the piano, so you will approach it like a piano. Yet, you don't have the mastery to make your fingers do everything you can think of. Your creativity is already confined by what you think the keyboard can do, and what your hands don't have the training to do."

The Valley of Mediocrity

At the end of the project, when everyone presented their pieces, it bore out almost exactly like he had said from the start. 

The output from the piano-proficient students was evident: they had lots of piano-isms, lots of notes, tended to be faster and very showy. 

The pieces from the third group were distinct as well: they were crazy, they were chaotic, they were creative and there was no influence of the keyboard-nature of the interface. They could have been using any bunch of buttons or other input device.

And the work of the middle group, including my own contribution I'm afraid, was as constrained as predicted (*cough* lame). Some broken chords, halting rhythm, simplistic little melody fragments, or in one case a full-blown song melody (picture: picking out the notes from a Disney theme song on the piano by ear). The pieces may have had some element of creativity underlying them, but then had been forced through conceptions of how keyboards should be approached, and executed terribly. One could hear the "sit up tall, two hands on the keyboard" tension more than one could hear the creative flow of musical expression.

Boxed In

Years later I would learn the concept of affordances in interfaces, and still continually wonder: when are I/we in that middle group?

The middle group was unable to get beyond their perceptions of the limitations of the interface (keyboards play notes, they play chords, they're played with two hands) and get to creative exploration of the whole range of possibilities (press them, smash them, roll a bottle over them). I could not see past the regular ordering of alternating whites and blacks (and years of piano teachers correcting my etudes) and thus my ability to create music was boxed in. I wanted to be in the masterful, first group, when the most creative work came from the unconstrained explorers in the third group. 

When do we let our perceptions of the limitations of our tools (the apps we use, the vehicles we drive, the mobile devices we carry, the models and frameworks we employ), prevent us from exploring their full range? Of thinking beyond the constraints we impose on ourselves before we even get started?

If only I could have forgotten how to play the piano for a semester and known nothing of the conventions of keyboards! My creations in that class would most certainly have been different (better? Maybe not, but different for sure).

Dr. Ben Smith is a Data Scientist and thinker, fascinated by the appearance of computers in our daily lives, creativity, and human struggles. He has had the privilege to think, learn, and write at the University of Illinois, the National Center for Supercomputing Applications, the Cleveland Institute of Art, Case Western Reserve U., IUPUI, and at Boardable: Board Management Software, Inc.

If you have feedback or questions please use Contact to get in touch. I welcome thoughtful responses and constructive critique.

The First Forms

When was the first web form put up on the internet? We live in a world of HTML and interactive web pages and apps, but how often do we stop and think about how quickly it is changing, and how little we really understand how it impacts how we buy and sell stuff?

While the first web browsers became publicly available around 1990, the first "sites" were static pages. The original HTML spec didn't include any of the "form" input type tags. It wasn't until 1993 that forms start appearing on pages. As of writing that makes the ability for websites to have fillable forms 28 years old. While we have decades, centuries of study on how to best design paper forms and how people use them, the web form is effectively a blip on the time line. We barely have time for Human-Computer Interaction practitioners to assemble and distribute insights into our behaviors before a disruptive innovation changes what we all do again!

But to access those early pages you had to have a dial-up modem that used your land-line phone connection, which was the only phone for the entire household (in the US mobile adoption wouldn't reach 85% saturation until 2008 (https://www.pewresearch.org/internet/fact-sheet/mobile/). At my house the internet connection was billed by the minute, equivalent to long-distance calls, and I repeatedly got in trouble with my parents for running up the internet bill! Broadband access was at 50% in 2000 in the US and only reached the 80% range in 2012. You could have the fanciest web site with a paid marketing budget, but until 10 years ago most Americans would not have access to find it.

The ability to effectively market a product on the Internet, and use the Internet as a primary method to reach buyers, is younger than my 6th grader! Search Engine Optimization (or SEO), the practice of carefully tweaking web pages to get ranked as high as possible in search results (and a critical job in online marketing), only became a going concern in the late 90s with the rise of Google (https://www.bluefrogdm.com/blog/a-brief-history-of-seo). In most of the "careers" that Americans talk about, and are taught at colleges and trade schools, we have a previous generation of practitioners to learn from. Yes, all careers are evolving in the age of Info Tech, but anything that is entirely Internet centric is entirely new. It's very exciting, and we just don't know where it will go!

Software-as-a-Service (SaaS)

Remember when "buying software" involved a stack of discs or DVDs that provided a tangible product you could hold in your hands? I knew I owned my copy because I had a shelf of discs I could touch. In the 90s it was inconceivable that one would buy software and have it delivered 'over the air,' let alone that one would pay a subscription fee for it. As recently as 2014 I received a weighty stack of discs with half a terabyte of software that took a day to install (only 8 years ago!).

The first company that is frequently cited is Salesforce which proved the viability of Software-as-a-Service by the early 2000s. Sometime around 2010 the market found a tipping point and SaaS becomes the dominant form of software development and delivery. I remember as one-by-one all of the professional software packages I owned converted from the DVD stack to a cloud based recurring subscription model. Particularly notable was the move of the Adobe Creative Suite to "Creative Cloud" in 2012/13. Did I "own" my copy any more? Did I ever, really, or was the ephemerality of software always destined to find this more ideal mode of distribution and possession?

Hiring Software: Jobs to be Done

If Software is a service industry then I no longer "buy" my apps, I "hire" them to provide a service (C. Christensen formulates this in his "Jobs to be Done" theory). I hire a music app to recommend new songs and set a good mood while I wash the dishes. I hire a map app to reliably get me where I'm going. I hire a suite of professional music recording apps, integrations, and plug-ins to enable my creative music making. And if any of them do a crummy job I "fire" them and find a replacement.

Not even 10 years ago (definitely younger than my 6th grader) does SaaS become the way apps are marketed and sold. With anything this revolutionary and different, how much do we still have to learn? A year feels like an eternity in this space, but looking back, we are in exceedingly novel and untrodden land. It feels like a wildly disruptive innovation could be around the corner at any moment. Whatever it is (VR? Ubiquitous smart watches? IoT?) It will provide even more new territory to explore, and opportunity for new products and solutions.

Through all of the brief history of consumer software the one constant has been us (yes, "software" has been around a few decades longer, but remember when "user interface" was a pretty novel concept? WYSIWYG editors have only been mainstream for about 2 decades).

People. We might spend a lot more time on our smart phone browsing social feeds than we could 20 years ago, but the underlying struggles we experience in our lives are not dramatically changed. We have the same fundamental issues (food, shelter, climate between ±40° C) that humans have always had. Maybe we can transpose our need for community to a distributed network of remote interest groups, but our social nature has always been with us. Now we have new things, SaaS and web forms, to try and address our common trials and tribulations. 

And who knows, maybe we'll make some beautiful things, learn more about ourselves, and transformationally address some of our societal challenges along the way. We can always hope!

January, 2022

Dr. Ben Smith is a Data Scientist and thinker, fascinated by the appearance of computers in our daily lives, creativity, and human struggles. He has had the privilege to think, learn, and write at the University of Illinois, the National Center for Supercomputing Applications, the Cleveland Institute of Art, Case Western Reserve U., IUPUI, and at Boardable: Board Management Software, Inc.

If you have feedback or questions please use Contact to get in touch. I welcome thoughtful responses and constructive critique.

Seeing Clearly

Experimentation helps us get a clearer understanding of how users find value in our product, what our users' needs are, and how we can best help them address those needs through our product. We want to understand the good and the bad, the positive and the pain as best we can so that we can keep improving our product and increasing the impact we have through growing our customer base.

Themes

In order to keep building towards 'big wins,' experiments should be set up to address one big theme (or several if the team has enough capacity to run several overlapping themes). The danger is that many, disconnected experiments cannot build on one another and will not paint a clear picture of how we are increasing growth. Focusing on one problem area helps Growth teams keep driving towards collective, bigger wins that can clearly show team success.

Winning

Any experiment that gives us insight into our users and our product is a success. The only 'failures' are inconclusive or badly executed tests. If we test a hypothesis and find out it is conclusively wrong we can learn and create better hypotheses!

A good experiment involves 4 steps: 

1. Developing a Hypothesis, 

2. Defining a Minimal Viable Test (MVT), 

3. Analysis, and 

4. Concluding and Communicating.

Assumptions

What is the one underlying assumption that must be true for your hypothesis to be true? Or, what is the one underlying assumption that if false makes everything else meaningless? Drive towards that assumption, because it is more likely to provide the level of insight that leads to 'big wins.' Lean into an inquiry model like the 5 Whys to dig deeper. Having clarity on the assumptions that underlie any hypothesis is critical to building a robust system of experimentation that can lead to meaningful improvements.

Developing Hypotheses

• Start from a problem. With a clear problem statement, brainstorm and develop best bet hypotheses to address it. Big problems (we are losing a lot of free trial users before subscribing), can be broken down into smaller problems (lots of users are leaving after 1 session; it's taking users too long to find value; too many users do not know what the product really does before signing up). If the problem statement has many issues combined, break it down into sub-problems. For each hypothesis keep the problem in focus and work from there with the goal of creating the highest value experiments.

  • Starting from a thing we know we can do will always be messy to test and prove, and increases the chance to be inconclusive. Be careful of starting from 'this will be easy to do' and working backwards to a problem. Similarly with measurement: 'this is easy to measure' and working backwards to a problem is likely to miss the real problems that need to be solved.

• One Solution for One Problem. A testable hypothesis should have 1 problem statement, and 1 hypothesis statement. Many hypotheses can be formed around 1 problem, and each will indicate separate tests. The problem and hypothesis need to be rooted in evidence (observations or data). The hypothesis should state an action to take, how it addresses the problem, and the positive result.

  • An example problem: "We see that 60% of people that start trial sign ups do not complete them." 

  • An example hypothesis: "Reducing the number of steps in the sign up process will make it quicker for people to sign up and increase the number of complete sign ups."

  • The null hypothesis we are trying to reject: "The number of steps in the sign up process do not significantly impact the sign up completion rate."

• Aggressively test the problem. A gentle test is more likely to be inconclusive and thus a waste of time and effort. We need to balance this with the potential risk, while being as aggressive in our tests as we can in order to discover the biggest wins.

• Value. How significant an impact will solving this problem have, potentially? If it will only increase X by 1 or 2 it may not be worth the time and effort, but if it increases X by 50% or 200% it could be worth a lot of investment and risk.

• Measurable. Do we have the tools ready to measure the impacts of the test, and can they be measured with an appropriate investment of effort? Can we be really confident that what we measure will be accurate? (For example: Google Analytics only catches 95% of traffic, at best, so it introduces 5% error in everything we measure there.)

• Trade-off metrics. Are there any potential negative impacts of carrying out this experiment? For example: making the trial sign-up process much easier will increase new sign-ups, but might increase the number of inactive trials (because casual users will be signing-up more frequently).

• Null hypothesis. In statistics we never prove our hypothesis conclusively (there's always a chance that some unknown other forces caused the differences that we see during our experiment), but we can often reject the Null Hypothesis: that our problem and our test are unrelated. For example a null hypothesis might be: there is no correlation between the text of the free trial sign-up buttons and the number of sign-ups. If we observe a strong correlation we can show that the null hypothesis is disproven.

Defining a Minimal Viable Test

• How minimal is minimal enough, but not so minimal that it is ineffective? We want to invest the optimal amount of time and energy in the test, but not any more than we need to (hence 'minimal'). The danger here is that we make it too light, without enough impact, to see any change. (The "a bigger button" type is rarely going to have the desired magnitude of impact.)

• Consider how it fits with other active or planned tests, and ensure that it does not conflict (in terms of influencing the data of other tests).

• Prioritize. Once defined, prioritize amongst all currently possible MVTs. Which one on the table is likely to give us the quickest, most valuable win? Is there an order to MVTs that will have the most impact (such as fixing problems that are earlier in the user experience to get more users further into the product where other tests can then have more impact)?

• Use ExperimentCalculator.com to get an idea of how many people/sessions will need to be sampled and how long the experiment will need to run.

Analysis

• One success metric. Pick one metric that will determine success. Any number of other metrics can also be watched for potential insight, but a clear single point will indicate if the test was conclusive.

• In small tests looking for the chosen metric to move in the right direction can be sufficient, but in other cases calculate statistical confidence and significance before closing the test and drawing conclusions.

Deciding and Communicating

• Decide. Based on the chosen success metric, decide to close the experiment at the point that a conclusion can be reached. Then make a clear conclusion: was the test conclusive, and if so, did the test move the metric in a positive or negative direction? Additional observations can be attached to this conclusion (if other metrics moved in similar or opposite directions).

• Communicate. The team needs to be proactive in communicating successes and successful test conclusions internally, to other teams, and the company. What we do is not obvious and clear to everyone and they will not see our outcomes in their day to day work. It is up to us to share what we are learning with the people who can benefit the most from it!

On the Dangers of Inconclusivity

In terms of the momentum of the team, and growing a culture of experimentation, inconclusive results can be very harmful early on. Inconclusive tests are par for the course, but without conclusive up/down results it is hard to communicate impact. And it is easy to become disheartened without clear impact. From experience, in order to build a 'test first' muscle, impacts and 'wins' are needed. See above on testing aggressively, and driving towards positive/negative outcomes that are conclusive.

Inconclusive results are like coloring in the background first, they're useful in eventually seeing the entire picture, but coloring in the primary character first is much more fun and encouraging! Any 5 year old will confirm as much. 🙂

May, 2020

Dr. Ben Smith is a Data Scientist and thinker, fascinated by the appearance of computers in our daily lives, creativity, and human struggles. He has had the privilege to think, learn, and write at the University of Illinois, the National Center for Supercomputing Applications, the Cleveland Institute of Art, Case Western Reserve U., IUPUI, and at Boardable: Board Management Software, Inc.

If you have feedback or questions please use Contact to get in touch. I welcome thoughtful responses and constructive critique.

Raising the average of a group of numbers presents 2 straight forward methods:

1. Add more numbers (more sales) to the high end, or

2. Remove numbers from the low end.

The second is the most problematic because the ASP will go up (yay!) but the rate in increase over all sales will decline (boo!). Selling 7 cars at $20K is not as good as 10 cars at $15K, in terms of total sales. See the previous image, where the average of the red curve (dotted line) is higher than that of the original black curve, but the area under the curves (representing total sales) is much smaller.

Another way to fool oneself into thinking an average is moving is to have a few values that are really, really far away from the mean. This would be selling 9 cars at $15K and 1 at $150K (ASP = $28.5K). The single outlier drags the average up, but by definition "outliers" are rare and not a regular observation in the typical population. This is a case where calculating an average is not appropriate, because the outlier (the Ferrari) isn't part of the usual set (a lot of Toyotas).

The clue that would tip us off is that the Mean Sale Price (MSP) would stay at $15K, while the Average Sale Price would jump to $28.5K. Relying on the MSP will be more reflective of what's actually in the data.

Problem 3: Multi-modal data

Perhaps sales are not 'normally' distributed, meaning they don't stack up in a nice bell-curve. At Boardable we sell 3 different levels of access at different price points, which can then be increased (buy more licenses) or decreased (discounts or special offers). This causes the spread of sale prices to clump around the 3 entry price points, making a tri-modal  distribution. Fitting these sale points into a single average gives us misleading results, because the data is not normally distributed:

In 'averaging' terms we are looking at 3 different populations, the entry level buyers, the mid-tier buyers, and the premium buyers. In many cases this basic split also reflects the types of buyers and what they're looking for in our product, and gives us more insight into how each 'package' is performing. See the next image, where each group is fit with its own average:

Solutions

1. Test for normality. At its simplest, load all your values into a spread sheet and look for a histogram chart. Eyeball it. If it looks like a bell or a mountain (see images on this post): go for it, ASP will be reflective of what's actually happening.

2. If it looks like multiple hills: cool! Figure out how to split it out into separate groups (populations) and determine what causes the clumping. Did they buy at a different time? Different packages? More or fewer add-ons or services? Is discount strategy identifying price sensitive populations? This is where insights will be uncovered.

   1. Then, it may be appropriate to have multiple ASP values that you track. Does each package move differently, based on business decisions or market conditions?

January, 2022

Dr. Ben Smith is a Data Scientist and thinker, fascinated by the appearance of computers in our daily lives, creativity, and human struggles. He has had the privilege to think, learn, and write at the University of Illinois, the National Center for Supercomputing Applications, the Cleveland Institute of Art, Case Western Reserve U., IUPUI, and at Boardable: Board Management Software, Inc.

If you have feedback or questions please use Contact to get in touch. I welcome thoughtful responses and constructive critique.

Dr. Ben Smith is a Data Scientist and thinker, fascinated by the appearance of computers in our daily lives, creativity, and human struggles. He has had the privilege to think, learn, and write at the University of Illinois, the National Center for Supercomputing Applications, the Cleveland Institute of Art, Case Western Reserve U., IUPUI, and at Boardable: Board Management Software, Inc.

If you have feedback or questions please use Contact to get in touch. I welcome thoughtful responses and constructive critique.