Emerging technology deserves a business test, not an automatic purchase or an automatic rejection. This checklist shows how to turn an interesting idea into a bounded experiment, measure it against existing work, and decide whether to stop, adapt or expand.
How Do You Assess Emerging Technology?
Start with a business problem, compare conventional solutions, and test the smallest useful application with explicit limits. Define success, unacceptable harm, total costs and an exit route before giving a supplier access to production data. A successful pilot supports a decision; it does not automatically justify a full rollout.
1. Define the problem and name its owner
- Write a problem statement. Why: “Explore AI” or “try augmented reality” describes a technology, not an outcome. How: Identify the affected people, the task, the current difficulty and the improvement that would matter. Keep the statement independent of a particular product.
- Assign a business owner. Why: Someone must decide whether the result is useful and accept responsibility for the workflow. How: Name a person who understands the process and can secure participation from its users. Give technical, security and purchasing colleagues clear supporting roles.
- Record the baseline. Why: Without a comparison, novelty can look like progress. How: Observe ordinary work, including difficult cases, and record the time, error types, rework and service constraints that matter.
For example, a maintenance company might want technicians to identify the correct replacement component without repeatedly calling a supervisor. Its problem statement should describe those interruptions, rather than assume that an AR headset is the answer. Better labels, an improved parts catalogue or a phone-based guide might solve the same problem.
Ask the people doing the work where the process actually fails. A manager may see slow completion while technicians see outdated diagrams. Those observations lead to different experiments.
2. Compare the technology with simpler alternatives
- Describe the relevant capability. Why: A broad category such as quantum computing contains very different propositions. How: State what the proposed tool would actually do, what inputs it needs and what output a user would receive.
- Build a small comparison. Why: The right decision may be ordinary software, a process change or no purchase. How: Include the current workflow and at least one simpler alternative alongside the emerging option.
- Separate demonstrated evidence from claims. Why: A polished demonstration may exclude your hardest cases. How: Request documentation, limitations and examples involving comparable conditions. Treat supplier projections as assumptions until your own evaluation supports them.
| Decision question | Evidence to collect | Warning sign |
|---|---|---|
| Does it address the problem? | Results on representative tasks | Demo measures a different task |
| Can our team operate it? | Skills, support and integration needs | Critical work has no owner |
| Is the cost acceptable? | Setup, usage, review and exit costs | Only subscription fees are quoted |
| Can we leave safely? | Export formats and deletion process | Outputs or configurations cannot move |
| Is it ready for this setting? | Reliability and known limitations | Research results presented as service guarantees |
A technology can be promising without being ready for your particular use. Keeping it on a watchlist is a legitimate decision when the capability, supporting tools or evidence is not mature enough.
3. Set boundaries for data, people and safety
- Map the information flow. Why: A small trial can still expose sensitive records. How: Document what enters the tool, where it is processed, who can access it, retention rules and whether a supplier uses it for other purposes. Prefer synthetic or appropriately de-identified material when that can answer the question.
- Identify affected people. Why: A workflow improvement for one team may create extra work or unfair outcomes elsewhere. How: Consider staff, customers and people represented in the data. Define where human approval is required and how someone can challenge an incorrect result.
- Set stop conditions. Why: A pilot needs protection against unacceptable outcomes, not just performance targets. How: Specify events that immediately pause the trial, such as unauthorised disclosure, unsafe instructions or repeated failures in an essential task.
For AI projects, NIST’s AI Risk Management Framework organises risk work around Govern, Map, Measure and Manage. Use that structure to connect ownership, context, evaluation and responses rather than treating the pilot as a purely technical exercise.
For broader security planning, the NIST Cybersecurity Framework provides an outcomes-based way to discuss cybersecurity risk. Neither framework is a product endorsement or a substitute for checking your own legal, contractual and sector-specific obligations.
4. Write the pilot plan before switching it on
- Choose a bounded workflow. Why: A broad rollout makes causes and consequences difficult to isolate. How: Select one process, a defined user group and a limited period. Avoid critical production decisions until the evidence and controls justify that exposure.
- Agree on measures and thresholds. Why: Teams can reinterpret disappointing results if success was never defined. How: Record the baseline, target improvement, minimum quality and maximum acceptable review burden before collecting results.
- Prepare recovery and support. Why: Experimental tools will sometimes fail or become unavailable. How: Keep the existing process available, identify a support contact and explain how users should report errors or return to normal work.
A useful pilot brief fits on a short document: problem, owner, participants, data, alternative, measures, limits, stop rules and decision date. Add a record of approvals where required. If the proposal cannot explain these basics, more preparation is needed before access expands.
In the maintenance example, the trial could compare the current catalogue with an AR overlay on selected non-critical training equipment. The plan would specify which component-identification tasks count, how incorrect identification is recorded and when users must consult a supervisor. It would not treat speed alone as success.
5. Run a fair comparison and capture hidden work
- Use representative cases. Why: Easy examples overstate usefulness. How: Include routine tasks, awkward inputs and foreseeable failure situations. Record exclusions so decision-makers know what the pilot did not establish.
- Track the complete workflow. Why: A tool may save one person time while adding checking or cleanup elsewhere. How: Measure preparation, correction, review, integration and support alongside the main task. Distinguish initial training from work likely to recur.
- Review failures with users. Why: A single overall score can conceal an unacceptable pattern. How: Group errors by cause and consequence. Ask whether users noticed failures, understood the tool’s limits and could recover without specialist help.
Suppose the AR guide helps on clean, well-lit equipment but becomes confusing around reflective surfaces. That is not merely an average-performance issue: it may define where the tool should not be used. The evaluation should preserve that distinction rather than hide it inside a combined score.
Keep a simple change log. If a supplier updates the model, interface or device during the trial, note the change and its timing. Otherwise, differences in results may be attributed to training or user behaviour when the product itself changed.
6. Decide, document and plan the next stage
- Choose stop, adapt or expand. Why: Continuing indefinitely consumes money without producing a decision. How: Compare findings with the original thresholds and risk limits. State what remains uncertain and whether another bounded test could resolve it.
- Calculate operating costs. Why: Pilot pricing and volunteer enthusiasm do not represent normal operations. How: Include licences, devices, connectivity, integration, staff review, training, monitoring, supplier management and eventual replacement or migration.
- Make expansion conditional. Why: More users and higher-stakes tasks introduce different risks. How: Assign ongoing ownership, support arrangements, incident handling and review dates. Reassess when data, users, use cases or supplier terms change.
A stop decision can be valuable if it prevents a larger commitment. Preserve the useful evidence: what failed, under what conditions, and what would need to change before reconsideration. Avoid converting a rejected pilot into an unapproved service simply because a few people still find it convenient.
Before expansion, check that export and deletion arrangements work in practice. An exit plan should cover data, configurations, documentation and the people who will operate the replacement workflow, not just contract cancellation.
Questions About Getting Started With Emerging Technology
Can a small business run a useful pilot without a specialist lab?
Yes, if the scope matches its ability to supervise the work. A narrow, low-risk workflow with clear comparisons can provide useful evidence. Use qualified help when safety, regulated decisions, sensitive information or complex integration exceeds the team’s expertise.
What if a supplier will not explain its technology in detail?
You do not necessarily need source code, but you do need enough information to assess intended use, limitations, security, data handling and support. If missing information prevents a responsible decision, restrict the test or choose another option.
How long should the pilot last?
Long enough to observe representative work and foreseeable exceptions, but short enough to maintain a clear decision point. Define coverage as well as a calendar end date. A quiet week that never encounters the difficult cases is weak evidence even if the trial ends on schedule.
Should staff know they are using an experimental system?
They should understand its status, limits, reporting process and approved uses. Explain what information is collected about their participation and follow applicable workplace and privacy requirements. Clear communication helps users report problems instead of treating the system as an unquestionable authority.
Begin With a Bounded Practical Trial
A disciplined pilot starts with a problem and ends with a documented choice. Protect the people and information involved, compare the whole workflow, and make further investment depend on evidence rather than excitement.
Next steps: Read what emerging technology means, review common evaluation mistakes, and explore the emerging technology hub for related guidance.
Sources
- NIST — AI Risk Management Framework
- NIST — Cybersecurity Framework
This guide is general education, not professional, legal or security advice.