How to Run an Effective Proof of Concept

Radar lists the top do’s and don’ts from technology experts.

How many hours? How much money? How many careers? How much hair? All lost forever, wasted, never to be re-generated. Why? Because some of the largest and best-resourced institutions in probably the fastest moving commerce vertical (financial services) somehow failed to follow simple golden rules when laying out a proof-of-concept (PoC) plan.

How tough is it to make a clear, well defined and achievable PoC that is intelligible to both the internal stakeholders, and the vendors that are attempting to become the provider of choice?

Well, actually the answer is “very tough”, and everyone has a different angle on how to do it. The vendor graveyard is littered with aspiring software giants that were broken by the mad tinkerings of demanding investment banks.

Without further ado, we will list the views of chief technology officers and industry luminaries and their do’s and don’ts to help institutions and vendor get to the right outcome with minimal collateral damage. Oh, and for good measure let’s start with the stellar team at Behavox (as they have the scars to prove it!)

Behavox Golden Rules for PoC success – Kiryl Trembovolski, chief operating officer, and Nabeel Ebrahim, head of account management

Scope out the requirements. What does the organization really need? Scoring must be applied to this approach and it is critical to ensure all relevant stakeholders give their requirements at the start; don’t allow anyone to come late and distort the process later.

The requirements must not be vague, lack clarity or specifics; they should not be subject to misinterpretation.

The Statement of Work must reflect this clarity. This is so important as without such definition the PoC cannot be closed out. Internal stakeholders cannot establish if the results delivered by the vendor satisfy the requirements.

The PoC team must have clearly defined, specific roles and responsibilities. What is the governance structure and decision making process? Is budget assigned and protected? Who holds it?

Be crystal clear on the desired outcome. Is it false positive reduction or increased true positive identification? Is it cost efficiency or organisational transformation? Avoid a desire for ‘sexy’ scope creep. Streamlining your lexicon analysis is certainly not sexy, but it is worth tackling before you get into amazing new analytics and cool behavioral stuff.

Next generation technology is all the rage but so many firms are not doing the simple stuff first.

Get your house in order and don’t specify something with an internal dependency that is not fit for purpose.

Watch for non-functional specifications. Beware the often yawning gap between what the business wants and what the IT team and the provider can realistically deliver, as the gap here is often considerable.

Define the environment. If this is testing against a legacy system, how many scenarios are to be tested? Is this on the full population or a reduced one for test purposes? If testing a vendor’s out-of-the-box scenarios against your own, find out how these were designed. Underline access control needs and explain escalation workflows.

For data, ask how much. How much historical production data or test data is relevant? How will the data be obtained and in what formats? How long will it take? Is there the necessary approval to use it? Can cloud be used so we don’t have the arguments around testing on own premises, procuring hardware and building servers? So many questions but all valid ones…

Define scope of integration works. Will the vendor need to integrate with your data systems as part of the PoC project? If yes, then internal system owners need to be identified and connected with the vendor. Ask the vendor to share relevant integration blueprints and dependencies on your internal IT. Make sure that there are no outstanding questions and assumptions on what integration works will be performed. Receive approval from IT for this.

Front-load the various departments to avoid blockers late in the process. Make them aware of the requirements, the progress and the vendors involved. Sometimes these interactions can be fatal, so better to clear these hurdles early to avoid wasted effort.

Be ready to sell to them the benefits of the new technology, as well as being transparent about the challenges.

Don’t underestimate the resistance to change internally and the fear of being made obsolete through technology adoption. It is usually a false fear, as technology generally empowers, revives and sustains the various teams that feel most threatened by it.

A vendor may not be for life, but it shouldn’t just be for one night either. Is it likely to be a five to 10-year partnership? Think about what you like/dislike about your existing vendor relations. It’s worth considering how much influence you wish to have with them over future product enhancement and design.

Marshall Wace’s chief technology officer – Conor Kiernan

If you are comfortable with the cloud, use it to your advantage. If you do a PoC “on prem”, costs can add up quickly in terms of the lead times, potential hardware expenditure, and the time spent by your engineers to build out the environment. The goal should be to run a PoC that can operate at scale and test the suitability of the solution for your business. If you are already comfortable with a cloud environment, using it can vastly speed up the PoC process and save on costs. Similarly, do not hesitate to use the expertise of the supplier to get a PoC environment up quickly, especially if the tech stack is unfamiliar to your engineers.

Too many cooks ruin the stew! Reduce touchpoints with the vendor to one. Reduce the business sponsor to one. Be aware that compliance, legal, tech and the senior managers will all have different angles and views, and so will want different use cases and features.

Run point testing on what is important. It is not realistic to build the world in a PoC. Technology is an evolution and the product will emerge as your needs change. Once you use and engage with it, you start getting better feedback. Many like to wait until the end and have this big brand rollout, but that is a mistake.

The sooner you make feature requests, the better. Get them in upfront not once installation is done, and get commitment to timelines and honesty on whether those will ever be enabled, or you could end up in a bad place. You want others to have success as well, to build an ecosystem to make the vendor sustainable so the product and company can survive. This means the product continues to be developed. It helps to get a priority for a feature if it helps everyone, not just you.

Work together. It can take time, but if you tweak the parameters or build a new model and see how that can accommodate new requirements, this is as important as if it worked right out of the box. In fact, I think this is even more important in that one person running the project and who is the touchpoint for the vendor should be meeting with internal stakeholders and keeping them informed so that they can evaluate and can input via that contact to prioritise and request appropriately.

Do not think about the past – think about the future. The client should not think about new software simply as a replacement for the existing functionality and workflows their current solution offers. New technology should pull you into the future! There is no better time to completely rethink and reshape your entire approach than when you are implementing new tech.

Ex-head of infrastructure change of TP-ICAP (now at Behavox ) – Shane Baker

Ensure the objectives are defined upfront. This must happen before any PoC is started or vendors are engaged.

Ensure the end users are brought in and have the right resources available. This is key so they can work on the PoC and provide it with the level of focus and interest it deserves.

Consider soft objectives. How are you going to assess the vendors’ ability to deliver? Having a good product is one thing, but can they actually deliver it in a production environment? What is their engagement like, assess the type of people they present, the speed of response, what pipeline of work do they have against the size of their organization?

Try to connect with a community of other users. There is no better way to improve the effectiveness of the platform and scenarios via collaboration, and this is an option for Behavox clients. Consider market, business and regulatory risk knowledge about the vendor.

How do you ensure it’s not just pre-sales hype and you are entering the void of fantasy software that the vendor cannot actually deliver in your required timeframe?

You have to be able to expose the vendors that are just slick at selling.

The view of PwC Financial Crime Technology Partner – Graham Ure

Getting consensus from the start on who the decision makers are. If the organization is seeking to make a global selection, ensure that you have all the right people to the party in the first instance. Lean towards inclusion rather than leaving people out.

We encourage clients to begin an evaluation with a long list of vendors. Even if it is only a representative list (though we understand this approach might be quite painful for the vendor!) to get a feel for what surveillance solutions are capable of. This then allows a level of upskilling in the organization. One of the challenges is moving an organization from the place where they want to evaluate a vendor against what they already have plus a little bit, to the place where they see the art of the possible and can therefore make an informed decision as to what solution to turn to that can future-proof their surveillance capability.

Forcing our clients to score at a macro and a micro level in the request for proposal (RFP). This is hugely important as it is a prerequisite for driving a level of objectivity.

There is nothing worse than finding you have a variety of different opinions after viewing a vendor solution but no agreed means to weight and score the importance of your requirements to make an informed judgment as to fit.

Vendors should demand a clear set of briefings and a well written Request For Information (RFI) in the first instance. Once the institution has gone to the trouble to articulate needs and given a view of existing infrastructure such as the APIs required and what systems you need to be able to link to, the vendor then gets a stronger sense of comfort that what they are entering into is real, and as importantly can provide a more tailored response.

Get the client to set up test data conditions. In an ideal world you need to know your own data and you need to know what is in that dataset that you want the vendor to be able to find so you can turn around and say Vendor A found 20 out of 22 things which we seeded and Vendor B found 10.

Bring a level of equivalence to the PoC. Ideally test conditions should be set up on the client site and should be the same for all vendors. Creating a level playing field from the start is important; all vendors should get the same amount of time within 9-to-5 to do their work and across a finite and consistent period. As a general rule vendors are asked to perform this for no charge and so there should be an onus on defining success criteria upfront and all parties should be clear on what those are