"You are crazy! A Proof-of-Concept (PoC) takes weeks or months to gain reliable insights. A hackathon is not enough time for providers to demonstrate how their products meet our needs. Hence, you can't make a reliable provider decision".
Well, we did it, and with excellent results. After three intense days of a Pocathon, we collected hard facts and insights to support the steering board making the right product decision. One provider - a leader in the Gartner Magic Quadrant - even gave up after the first day, telling they don't provide what we need. Epic! Ultimately, we reached a consensus on what product to choose despite prior biases and corporate politics.
In the following, I'll explain what the Pocathon is, the benefits, and the preconditions. Then, I'll explain the Pocathon approach from preparation to execution, to assessment. It will help you to run a Pocathon yourself. Finally, I provide some best-practices to run a successful Pocathon because the devil lies in the details.
What is a Pocathon
A Pocathon combines the advantages of Proof-of-Concept (PoC) and hackathon and both terms. With a PoC, you test how well your organization and requirements fit with providers and products. With a hackathon, you achieve quickly many learnings and gain first-hand insights beside polished sales pitches.
Providers form teams who apply a product to meet your needs. The teams participate in a hackathon-like event to do the PoC. Teams arrive, sit in one room together with users to solve the use cases and eat Pizza ;) Finally, they'll pitch their solution to the users, stakeholders, and decision-makers.
The benefits of Pocathon are:
Fast: In a short period, multiple providers and products can be evaluated and compared. Pocathon provides just enough insights to make the right decision.
Cheap: Preparation, e.g., infrastructure and execution are out-sourced to the providers. The investment of providers is only a few days compared to weeks or months for a classical PoC.
Efficient: Strict focus on the scope and short time-boxed Pocathon. It provides practical, first-hand insights on sweet spots and weaknesses of products and how experts use them - may provide even blueprints. Further, it provides insights into the provider's culture, attitude, and internal decision processes.
Fair: Providers and products get compared based on the same scope and use cases. There are no biases like scope changes.
Stakeholder involvement: Pocathon is a participative process with users and stakeholders. Their involvement leads to a higher acceptance of the product decision. Further, they provide you relevant info and perspectives.
What's the Problem with Classical PoCs?
Classical PoCs are intransparent and unfair. Also, the lead time is high, typically weeks or months. Imagine that you invite the first product vendor. In the first few weeks, he'll wait for all the required access and firewalls to become open. Then, he'll get some fuzzy requirements. We achieve learnings from this PoC, discover new needs, and change the scope. A next PoC will be not comparable to the previous one. As a result, we do many PoCs, change requirements, and ultimately, the last PoC wins because it matches best our latest needs.
The common PoC anti pattern: many PoC are done, scope change, lots of effort and time passes, and the last provider wins anyways.
Why is a Pocathon Just Enough?
The Pocathon reduces the risks of an inferior product decision. It explores and enables the next step quickly, i.e., it's an exploration enabler story regarding SAFe. That's the agile approach. Further, it removes the overhead that doesn't reduce the risks. That's the lean approach. In other words, Pocathon is a lean, agile evaluation approach to make the ultimate product decision.
Pocathon is a lean, agile evaluation approach to make the ultimate product decision.
The Pocathon helps to compare integration efforts among products, no effort estimation, though. Pocathon helps to identify the best options, not how much it will cost. Let's be lean and mean ;) If it turns out in a minimum-viable-product (MVP) that it's too expensive, go back and choose the next best alternative.
The Four Stages of a Pocathon
The Pocathon consists of four stages: Scope and providers, preparation, Pocathon, and assessment.
Scope and provider: Define the scope and requirements what you are looking for. Construct the use cases accordingly to test products. Gain insights and evaluate who provides a feasible solution.
Preparation: Prepare the Pocathon days to run it smoothly. Nothing should negatively impact the teams' performance.
Pocathon: Run the pocathon, test the products, and gain insights.
Assessment: Assess the results and provide hard facts and insights to decide.
We run the Pocathon from preparation to execution, to assessment in two months. The following image illustrates our process with the four stages.
In the following, I'll describe the four stages in more details.
Stage 1: Scope and Provider
The scope of what you're looking for is vital, especially what you're not looking for. Don't get trapped considering irrelevant features. Run workshops to involve stakeholders and users and to collect needs and requirements.
Describe the responsibilities of every component, the ones that are in scope, and the ones that aren't. It helps to argue why a feature is relevant or not with stakeholders and with providers. The clearer you get, the simpler the assessment.
The purpose of the use cases is to test how the product meets the requirements and to gain insights. The use cases need to be realistic enough to set the context right. You gain nothing if you test an assessment criteria twice. For instance, you gain no info when a product has to call two similar APIs.
Define challenges and give them teams when they achieve a particular milestone to see how the product might support decision making and how flexible it can be adapted. Select challenges for use cases wisely to gain maximum insight. Note that we asses the product's qualities and not which team has the fastest engineers. Reduce to the max.
Cover your assessment criteria in your use cases and challenges as much as possible. Whatever you can't cover in the use cases, assess it with a workshop or one-on-one interview with the provider.
Stage 2: Pocathon Preparation
You'll need one room to host all teams with sufficient infrastructure (e.g., Internet, electric power). A single room makes it easier for you to move from team to team, be available for questions, and overview the overall progress. Consider a second room for interviews/workshops and probably a larger third room for the pitch that can host all stakeholders, users, and top management.
Fix the dates as fast as possible with the providers. The providers require time to get the resources for the Pocathon. Some providers will argue that it's short notice. Well, this will indicate their flexibility, commitment, and internal decision processes. The way they sell indicates the way they deliver.
Provide a simulation environment if you need to see a product interact with peripheral systems. Real integration adds unnecessary overhead. Just check for relevant integration capabilities. As the simulation environment, we used Node-RED to provide APIs, simulate backends, and even trigger events. We used its dashboard functionality to test and monitor the solutions.
Stage 3: Pocathon Execution
The Pocathon is intense, especially for the teams and for you. Scaling is an issue. You need at least one core team member per two teams. The member helps teams to be successful, answer questions, and gain first-hand insights. Ideally, the core team member has engineering and domain expertise.
Here's our agenda for a three-day Pocathon:
Day 1: The teams will need time to understand your business vision, product need, use cases, and explore the simulation environment. Use the first day to align the teams on what you are looking for. Day 1 sets the context. Consider providing only one use case to not overload teams with information. Finally, do a roundup, check the progress, and adapt the next day accordingly.
Day 2: Teams have progressed. Get first-hand impressions on the products and the developed solutions. Stay with one team for a while to grasp the strengths and weaknesses of the product. Collect info for the assessment. Finally, do a roundup, check the progress, and adapt the next day accordingly.
Day 3: Collect the missing info for the assessment. Run interviews regarding target architecture fit, integration effort, etc. You may provide the teams with your questions in advance to prepare. A period of 1.5 hours is reasonable. Invite relevant stakeholders to these sessions. Finally, let teams demo how they solved the use cases. A period for a 30min pitch and 30min Q&A is reasonable. The pitches will give the stakeholders an impression of the product's qualities. Use your insights from the previous days to ask the right questions that will uncover the good and the bad parts. Finally, celebrate together! All participants deserve it.
Stage 4: Pocathon Assessment
After the Pocathon is before the decision making. Assess the products, solutions, and providers quickly after or even during the Pocathon when your insights are fresh. Add comments to your assessment to provide the arguments for later discussions.
We graded every assessment criteria on a 1 to 5 scale. We classified the criteria into five categories: functional, technical, strategic, integrational, and cultural.
Functional Criteria: It covers all functional aspects that are relevant to meet the requirements. How comfortable and straightforward are design and development? It also includes how well the product solved the use cases and adapted to challenges.
Technical Criteria: It covers non-functional aspects, e.g., scalability, reliability, and extensibility. How well does it fit agile development?
Strategic Criteria: It covers commercial aspects, e.g., costs for licenses, infrastructure, operations, and development. What are the required skills and workforce? How well scales the maintainability?
Integrational Criteria: It covers relevant integration patterns and technologies aspects (e.g., REST APIs, real-time streaming).
Cultural Fit: It covers soft aspects and summarizes your observations and interactions with providers, e.g., engagement and flexibility.
Everything is a test! Add some stress into the mix. Then, you'll also observe how providers cope with uncertainties and short-notice changes.
Pocathon Best-Practices
Form a core team that consists of relevant stakeholders who add value. The members bring the stakeholders perspective and champion the Pocathon process.
Have an alignment on the scope in the core team and the steering board. What are you looking for, and what not? If the scope is fuzzy, so will be the foundation to decide.
Minimize the providers' efforts. If the efforts are too high, providers don't participate. Fit a Pocathon into three days or less and two people per team at most.
Don't over-engineer the assessment criteria - the fewer, the more understandable the final result. Avoid weighting criteria to get biased scores. Let data drive your discussions and do not leave the responsibility for the decision to your data.
Complement use cases with challenges and interviews to gain all relevant insights. Throw prepared challenges to the teams when they reached a particular milestone. Observe how they draw conclusions and adapt their solution.
Conclusion
Pocathon is a novel approach and initially invented by my colleague Dominik Liebman and Konrad Durrer. Pocathon allows evaluating providers based on a lean, agile approach. It's fast, cheap, efficient, and enables to compare providers more fairly and unbiased.
It is vital to have a clear understanding of what is in scope and what not. Either you work it out beforehand or invest time and effort with traditional evaluation and procurement approaches.
Everybody involved is now enthusiastic about Pocathon and would do it again. The critics became fans. The participative process involves relevant stakeholders and users, which brings higher acceptance. Most importantly, Pocathon provides hard facts to make an informed decision.