Emerging technology projects often fail between an exciting demonstration and the ordinary conditions of daily work. This guide explains the evaluation mistakes that create that gap, with practical fixes for pilots involving AI, augmented reality, quantum-related services or other developing technologies.
What Emerging Technology Mistakes Should You Avoid?
The central mistake is treating technical novelty as evidence of business value. Avoid it by defining a problem, comparing simpler alternatives and testing realistic tasks within explicit cost and risk limits. Keep a genuine option to stop, and do not confuse a successful pilot with proof that deployment will work at scale.
Mistake 1: Buying a technology before naming the problem
Why it hurts: A request to “do something with AI” encourages teams to search for uses that justify a preferred purchase. It can divert attention from problems that need better processes, clearer information or ordinary software rather than an emerging capability.
How to fix it: Describe the operational problem without naming a technology. “Technicians need to find the correct maintenance instruction more quickly” is testable. “We need augmented reality” is already a solution choice.
List the people affected, the current process and the consequence of leaving it unchanged. Ask whether the problem happens often enough to justify evaluation and whether anyone has authority to change the workflow.
Then compare options, including doing nothing for now. A searchable manual, improved labelling or staff training may solve enough of the problem with less integration work. An emerging technology should earn its place against these alternatives rather than receiving a separate standard because it looks innovative.
Mistake 2: Treating publicity as a maturity assessment
Why it hurts: A technology can be well known without being reliable for your task. A research milestone, product announcement and production-ready service represent different kinds of evidence. They do not establish the same level of availability, support or operational fit.
How to fix it: Separate capability from readiness. Ask what has actually been demonstrated, under which conditions and by whom. Distinguish a provider's claim from a reproducible result and from your own controlled evaluation.
| Claim you hear | Evidence to request | Question still open |
|---|---|---|
| “Works on complex documents” | Results on representative documents | Does it handle your formats and exceptions? |
| “Enterprise-ready” | Access controls, support and operating documentation | Do these meet your requirements? |
| “Easy integration” | A working connection to a relevant system | What maintenance will your team own? |
| “Quantum advantage” | Task, comparison method and limitations | Does that result apply to your business problem? |
Treat hype cycles as reminders that expectations fluctuate, not as a calendar predicting when you should buy. Reassess maturity for the specific application, not for the technology category as a whole.
Mistake 3: Running a demonstration instead of a test
Why it hurts: Demonstrations usually emphasise a favourable path. They may omit difficult inputs, recovery steps, human corrections and the time spent preparing the result. A polished presentation can therefore conceal the work that deployment would require.
How to fix it: Test representative tasks, including ordinary exceptions. Record the full workflow from input preparation to review and final output. Use the same task boundaries when comparing with the existing process.
For an AI drafting assistant, measure the time needed to produce an acceptable response after checking facts and editing, not merely the time until the first draft appears. For an AR aid, include setup, visibility, comfort and fallback instructions when tracking fails.
Choose acceptance criteria before reviewing results. Keep some evaluation cases separate from the cases used to configure the system. Otherwise the team may inadvertently optimise for a familiar demonstration rather than the broader task.
Involve actual users, but do not interpret enthusiasm as proof of effectiveness. Ask them to complete work and explain where they lost confidence or needed help.
Mistake 4: Moving sensitive work into the pilot too soon
Why it hurts: “It is only a pilot” does not prevent information leakage, incorrect decisions or harmful reliance. Small experiments can still process real personal data or produce outputs that affect customers and employees.
How to fix it: Define permitted data and actions before granting access. Start with synthetic, public or appropriately prepared examples where feasible. Check whether removing names is sufficient; distinctive details may still identify someone.
The NIST AI Risk Management Framework organises AI risk work around Govern, Map, Measure and Manage. Use that structure to identify affected people, evaluate potential harms and assign oversight rather than assuming accuracy is the only risk.
For any technology, identify who can pause the test and how problems are reported. Keep consequential decisions under suitable human authority. A reviewer needs enough expertise, time and information to challenge the output; adding an approval button alone is not meaningful oversight.
Document prohibited uses, including autonomous actions outside the trial's scope. Revisit the assessment if the vendor changes the model, hosting arrangement or data handling terms.
Mistake 5: Ignoring integration and the full operating cost
Why it hurts: A licence or usage charge is only one part of deployment. Data preparation, connectors, staff training, review, monitoring, accessibility and incident handling may dominate the practical effort. A useful standalone tool can become an awkward extra step in a real workflow.
How to fix it: Map what happens before and after the new tool runs. Identify required systems, data owners, authentication arrangements and exceptions. Estimate operating effort using clearly stated assumptions, then revise those assumptions from the pilot.
For example, a hypothetical document tool saves five minutes of drafting but adds seven minutes of verification and formatting. Its generation speed has improved while the end-to-end task has become slower. Investigate whether better workflow design can change that result; do not simply report the five minutes as a saving.
The NCSC cloud security principles provide useful prompts when a service involves cloud delivery, including identity, administration and protection of data. They support evaluation; citing them does not certify a supplier or guarantee that your configuration is secure.
Mistake 6: Letting the vendor define success and control the evidence
Why it hurts: A supplier understandably wants its product to succeed, but its preferred metric may not match your outcome. Results can also become difficult to verify if logs, test settings or exports are unavailable.
How to fix it: Keep a buyer-owned evaluation plan. Define the task, baseline, success measures, risk limits and test cases independently of the sales presentation. Ask the supplier to explain limitations and support your measurements, not replace them.
Preserve enough records to reproduce important findings, while respecting privacy and contractual constraints. Record product versions and configuration changes. If performance shifts after an update, those details help separate a changed service from a changed workload.
Check exit arrangements early. Can you export necessary data in a usable format? What happens to integrations, stored content and user accounts when the trial ends? Avoid assuming that data deletion and access removal happen automatically.
Commercial terms should be reviewed by appropriate colleagues. An attractive pilot arrangement may not describe the costs, responsibilities or service expectations of routine operation.
Mistake 7: Scaling because stopping feels embarrassing
Why it hurts: Once leaders have publicly backed a project, neutral results can become politically difficult to report. Teams may keep extending the pilot, changing measures or expanding scope until something looks positive. This converts learning into a search for justification.
How to fix it: Agree stop, revise and proceed conditions before the trial starts. Treat a well-supported decision not to deploy as useful evidence, not a failed team. Keep sunk spending separate from the future costs and benefits of continuing.
A pilot might show that a tool helps a narrow document type but fails on others. The appropriate decision may be a restricted deployment, not an organisation-wide launch or complete rejection. Document the boundary clearly.
Before scaling, reassess support capacity, security, reliability and ownership. A small group of enthusiastic users can absorb manual work that hundreds of users cannot. Scale introduces new questions rather than merely repeating the pilot with more accounts.
Questions About Emerging Technology Mistakes
What if a senior leader has already promised a launch?
Separate the promised ambition from the evidence needed for safe deployment. Present a bounded evaluation with explicit decision points and explain the consequences of bypassing them. Escalate unacceptable risks through the organisation's normal governance channels rather than quietly weakening the test.
Can a free pilot still create significant risk?
Yes. Staff time, data exposure, integration effort and dependence can arise without a purchase price. Apply proportionate controls to what the tool does and accesses, not merely to what the trial costs.
Should a disappointing pilot rule out the whole technology?
Usually not. It may rule out a provider, configuration or use case under the tested conditions. Record those conditions so future teams can distinguish a genuine capability limit from an implementation problem or an unsuitable task.
How do we handle a supplier that will not disclose limitations?
Treat the lack of usable evidence as an evaluation finding. Reduce the scope, require additional safeguards or stop if the uncertainty is incompatible with the task. A claim that details are proprietary does not remove your responsibility to understand operational risk.
Assess Fit Before Adopting New Technology
Good evaluation protects a business from both fashionable overinvestment and reflexive dismissal. The aim is a defensible decision about one use case, supported by evidence and a workable exit.
Next steps: Read the emerging technology overview, run the practical evaluation checklist, and explore the emerging technology hub.
Sources
- NIST — AI Risk Management Framework
- UK National Cyber Security Centre — The cloud security principles
This guide is general education, not professional or security advice.