Understanding Roobrick

Since writing this blog post, we've created an explainer video video for Roobrick. It's a great supplement to help understand Roobrick.

Think back to the first time you came across a specialised application such as a Customer Relationship Manager (CRM), issue tracker or Content Management System (CMS). It probably took you a while to understand some of the basic concepts and how they are implemented by the application to achieve certain goals.

Roobrick is also a specialised application. It is centred around concepts that are not widely used in certain circles which means it may be difficult for some to comprehend it.

This post explains the concepts that Roobrick is based on and the benefits it provides to individuals, teams and organisations.

Qualitative Assessments

Qualitative assessments occur everywhere. Many decisions involve evaluating the best option that meets a range of qualitative and quantitative criteria. For example, which CRM solution meets a teams’ needs best? Other decisions are of a yes/no nature such as should we invest in startup Xyz?

Decisions such as these necessitate the consideration of a large number of factors, many of which are qualitative. For example, do the founders of the startup have the experience and drive to succeed?

There is also a huge amount of qualitative evaluation that necessitates a grade rather than a decision. How healthy is our team? How well written is a given OKR? In the case of the OKR example, we need to prevent a team spending months of work striving to meet goals that lack clarity or are not based on available credible data, so we must ensure the OKR is defined well before the work starts. Various factors need to considered such as whether the objective expresses goals and intent, whether key results are aspirational and measurable and whether the attainment of the key results will attain the objective.

Evaluations such as these occur in all businesses and all industries. Best practice evaluations yields both rigour and speed and can make a huge difference to the success of a business.

Substandard Assessments

All too often, qualitative assessments are made upon incomplete and/or unclear criteria. In other cases, problems are not solved with the rigour that could be attained by breaking them down and establishing evaluation criteria.

Let’s consider analysing the best house to buy. A poor decision would be based on considering only basic criteria such as size and cost, landing on a choice based largely on emotion or bias. A more robust decision would factor in a range of other criteria such as proximity to transport, layout, etc. Whilst this may be an over simplified example, this kind of incomplete analysis is common throughout many businesses.

Sometimes we elude ourselves into thinking we are considering a range of criteria when we are not. This situation arises in various circumstances, the first being when we lump together pros and cons. This is primarily because it is very easy for each set of pros and cons to address different sets of criteria for each option since the presentation format does not force it. Further, choosing between a pro and con is a very course way of classifying the disposition of a given aspect of a decision.

The second situation where we allude ourselves about the rigour of our analysis is when we solve problems by creating detailed documents comprising sections discussing each consideration separately, resulting in an essay or report. Whilst this form of analysis presents and reads well, it lends itself to bias and often amounts to an argument for a particular stance.

Substandard assessment methodologies results in cognitive biases such as anchoring, loss aversion and confirmation bias which in turn leads to misguided results.

Empiracle research shows poor decision making is common in many organisations. For example, in a survey by McKinsey only 20 percent of respondents say their company excels at making decisions. A separate McKinsey paper finds ineffective decision making costs an average Fortune 500 company roughly $250 million in wasted labour cost.

To rectify these issues, we need to introduce better practices and bespoke tooling. This is the motivation behind the development of Roobrick.

Specialised Assessment Applications

There are a number of scenarios where assessments are made with a great deal of rigour and/or speed. Typical examples include a bank determining the amount they are willing to lend clients or insurance companies determining policy costs. These assessments are handled by specialised applications and processes since they are repeated many times and involve complex calculations.

However, we also need to be able to make swift, yet robust assessments for less common scenarios, including those that are “one offs”.

In the same way that a CRM application can manage any set of customer relationships, a generic assessment solution necessitates a specialised application that can handle any assessment.

A Generic Approach to Robust Assessments

It is reasonably straight forward to define a generic approach to making robust assessments, omitting process details that deal with organisation specific peculiarities such as stakeholder identification and approval requirements.

A generic approach for making a robust assessment starts with establishing clear criteria relating to the problem at hand. The approach can be used in a range of contexts such as personal circumstances, specialized problem domains, team circumstances or organisation standards. Here are some example assessments for each:

Personal circumstances: Choosing a university course
Specialized problem domains: OKR writing guide assessment
Team circumstances: Prioritising projects
Organisation standards: Job candidate assessments

Returning to the house buying example, obvious criteria would involve price, proximity to transport, proximity to work, number of bedrooms, etc. It is also necessary to define some sort of scoring system for the criteria and the interpretation on a per criterion basis. A simple scoring system may involve three levels such as “bad”, “ok” and “great”. A couple with one child may specify the “number of bedrooms” criterion as “1 = bad”, “2 = ok” and “3 or more = great”.

The effort invested into establishing criteria depends on the importance of the assessment and how often the assessment will be repeated (if ever). Once the criteria defined, it can then be used evaluated to form one or more rigorous assessments. In the case of buying a house, each house considered would require a unique assessment against the criteria. This is the well established practice of using rubrics.

What is a Rubric?

In general, a rubric is an evaluation guide, expressed as a set of criteria, guidelines and/or behaviours that specify how to evaluate something to form an assessment. Rubrics are widely used in academia since they:

Provide a comprehensive, repeatable and shareable guidelines for evaluating assignments.
Serve as a communication guide between teachers and students.
Result in more consistent assessments by different assessors and separation of assessment time.

Rubrics

Rubrics are typically presented in tables. Large tables can be unwieldy to navigate so Roobrick has some tricks to overcome these challenges. For example, scoring guideance often requires lengthy amounts of text to explain expected behaviors, etc. Roobrick allows guidance to be hidden and dispayed only when needed, resulting in a much more succint presentation.

Assessments

An example visualisation of a Roobrick assessment

Whilst rubrics are the central artefact of Roobrick, an equally important artefact is the assessment. Roobrick allows any number of assessments to be made against a rubric. For example, different assessments can be made by different individuals or teams. Alternatively, a single person or team may use a rubric to assess multiple things.

Visualisations of assessments provides insights that may otherwise be missed. This is especially true for rubrics that are structured with areas and labels since they provide additional dimensions to view the assessment by. For example, a rubric detailing expectations of a role at various career stages may be structured as follows:

Areas: technical, collaboration, outcomes
Labels: impact, proficiency, motivation, leadership, learning agility

This allows aspects of an assessment to filtered. In the above example, the future impact of an employee could be gauged by concentrating on the considerations only relating to the labels motivation, leadership and learning agility.

System of Record

The heart of SalesForce is the data it manages for its customers. As such, SalesForce is said to provide the system of record of sales information. The central concepts of Roobrick are rubrics and assessments. Thus, Roobrick provides the system of record of structured analysis.

To understand this, let's say a company uses Roobrick for assessing job candidates, team health, making decisions, etc. Over time, the company will build a suite of rubrics providing best practice guidelines and criteria for a range of processes. In addition, the company will define one-off rubrics for specific circumstances such as unique decisions. Mining historical structured structured analysis allows a company to improve future decision making and problem solving. For example, let's say a company defines a rubric for hiring new employees and an assessment against the rubric is made for each job candidate. At any time, it would be simple to compare the assessments of successful job candidates with their first performance review after being hired. Themes from the comparison may identify patterns such as a correlation between high assessments in some areas of the rubric with strong performances. This analysis may lead to further improvements of the hiring rubric.

Since the company extensively uses Roobrick for its future and historical structured analysis, it follows that Roobrick provides the system of record of structured analysis.

Collaboration

Rubrics within Roobrick can be shared with individual contacts and at an organisational level. In many cases it is desirable for rubrics to be shared openly. For example, any business can benefit from a rubric that specifies criteria determining how well a KPI or OKR has been written or a rubric that ascertains the health of a team. It may be appropriate for other rubrics to be only available within an organisation such as grading job candidates or tactical rubrics defining criteria relating to a specific decision.

Roobrick assessments can also be shared, however, assessments tend to be more private than rubrics since they generally pertain to more specific circumstances. For example, a public rubric may exist that sets criteria for measuring the health of a project team, but a given team may not want to share their self assessment against this rubric especially if annotations within it are particularly candid.

Re-usable Rubrics

Roobrick allows rubrics to be re-usable which is particularly useful for repeated assessments against the same criteria. Job candidates and team health are examples of recurring assessments against unchanging criteria.

Allowing rubrics to be re-used encourages investment in the quality of rubrics which further increases the robustness of the assessments.

There are also circumstances where the criteria only partially changes. To address this scenario, Roobrick allows rubrics to be copied and edited. This may be the case where best practices are revised or when a decision needs to be made that has similarities with a previous decision.

Integrations

Integrations with Roobrick allow its rubrics to be used within other contexts. For example, a certain type of Jira issue may be classified using a rubric. Alternatively, a rubric may be embedded into a document such as a Confluence page or used to assess items in a monday.com board.

Who Can Benefit from Roobrick?

Despite being thought of as an academic tool, rubrics appear everywhere. If any of the following scenarios are applicable to you, then you can benefit from Roobrick:

Make robust decisions by assessing different options against a set of criteria.
Manage the career growth of employees using guidelines for successive job levels.
Assess the health of a team using guidelines in health monitors.
Assessing how well a KPI or OKR is written (conformance against best practices).
Job candidate assessments against company or team requirements.
Assess a customer facing feature (and potentially share the assessment with customers, inviting customers to create their own assessments).

Roobrick in Action

Enough with the theory. Have you every investigated putting solar panels on your roof? There are numerous technologies and site specific considerations that make this a challengin task. The following rubric defines all the criteria you should be thinking about. It can be used to assess supply and installation quotes against one another to help you choose the right system for your circumstances:

View an example rubric: Residential solar power system choice

You may also like some of our other posts...