By breaking down the process of technology assessment into a series of executable, repeatable, and documented steps, we can be more informed and confident with our selection of technology.
The implementation of a telehealth system, or technology, will require a team approach. Team members will need to address issues that are equal parts clinical, technical, operational, and systems related. From the very beginning of the technology selection process it is important to include the insight and viewpoints of a variety of stakeholders. Many of the required areas of expertise may already exist within an organization, with clinicians, administrators, case managers, technology, and networking staff able to provide input.
Often, instead of a thorough technology assessment, organizations simply select a device by asking clinicians to choose the devices that they feel might meet their needs, or by asking IT staff to choose the equipment which they are able and willing to support. Interdepartmental teams may be built, vendors may be brought in, and a decision may eventually be made, yet a standardized and documented process for comparing and selecting technology was never implemented.
There are two likely outcomes of this slightly-haphazard process: organizations build up expertise over time and gradually develop a process that is used for selecting equipment, or else they make purchasing decisions that eventually fail to meet their needs. Based upon feedback from various telehealth programs, organizations rarely have the time or staffing to reach the first outcome, and the most common outcome is (potentially) expensive misplaced technology decisions.
At its most benign, a failed technology selection may result in a product that works in a telehealth environment, while delivering an experience that isn’t as polished as might exist with other products. Perhaps the device is a little hard to use, or perhaps the data being gathered or transmitted is simply good enough, rather than great.
Some of the more critical potential outcomes are technology selections that result in failed pilots, delayed programs, system incompatibilities, clinical user dissatisfaction, increased programmatic expense, and at their worst, harm to patients. Organizations clearly want to avoid these negative outcomes, but many don’t quite know how to do so.
The goal of this toolkit is to provide organizations with a basic understanding of the work that should go into a technology assessment so that a consistent and repetitive process can be put into place when selecting technologies. The processes and procedures recommended here will hopefully assist in the decision making process, helping an organization’s interdepartmental teams make choices that meet needs, extend the reach of their programs, and produce results that work well into the future.
That said, less-than-ideal purchases may still be made. By following a standard guide, however, the impact may be reduced, and an organization may be able to establish where and why the decision was made and how the process can be improved. TTAC strongly recommends that organizations continually refine their policies and procedures to meet their needs as they gain more experience in technology assessment.
Process Overview
Each technology assessment you perform will vary due to differences in technology, organizational competencies, time constraints, and available resources. In general, however, a thorough technology assessment should include these phases and steps:
Needs Assessment– Understand the needs of your program and the solutions that might meet them.
- Step 1: Establish Requirements
- Step 2: Market Review
Testing– Create, execute, and communicate a repeatable testing process that evaluates devices against your organizational needs.
- Step 3: Procure Devices
- Step 4: Plan Tests
- Step 5: Test the Plan
Deployment and Support- Purchase, deploy, and support selected devices.
- Step 6: Select Device
- Step 7: Deploy
The following sections will provide a high-level description of the above steps.
Phase 1: Needs Assessment
The decision to perform a technology assessment doesn’t happen in a vacuum. We generally are supporting a decision to implement a new telehealth system or program, or to expand existing telehealth services. A clear understanding of what needs we are trying to address, and how exactly we expect a device or solution to address those needs should be the first step in any assessment process.
Step 1. Establish Requirements
It will be useful to take time prior to beginning the actual assessment of the technology to perform an assessment of your organization’s needs. These needs should be documented in a list of equipment requirements that can be used in the assessment process. What clinical needs will the device or devices have to meet? What patient population are we going to serve with this deployment? What workflows will need to be changed to implement the technology successfully? What buy-in do you have from clinicians, administrators, and technical support?
A thorough assessment should look beyond the basic need, such as “we need to do tele-dermatology,” to find more meaningful organizational needs. A SWOT (Strength, Weakness, Opportunity, Threat) assessment may help in this process. Additionally, performing a SWOT assessment can help you determine which resources you have available within your organization that will assist in the review, selection, implementation and support of this technology into the future.
Read more about needs assessment.
Step 2. Market Review
Once your organization has a firm grasp of its needs and requirements, it is important to spend some time looking at what the market can offer. Some organizations will struggle with this particular step, as it can be hard to understand what differentiates one product from another, and what features are truly important to your program. Dedicating time to learn about the market can help your organization make informed purchasing decisions later in the process.
Gathering information can be challenging with many technologies used in telehealth. Each manufacturer will often use different terms to describe similar functionality, making it hard to establish any level of feature parity between devices. Additionally, some devices used in telehealth may come from a consumer-focused market space, which may emphasize features that are not pertinent to delivering telehealth.
Discussions with vendors and manufacturers should be balanced with independent research into the technologies, so as to avoid some of the bias that occurs when talking to people with a vested interest in selling equipment.
A market review should include a comparison of features between devices, with an emphasis on features that meet the criteria identified as requirements. The output of the feature comparison should be a document that lists the potential devices on the market, including a tabularized comparison of all of the devices.
The second aspect of a market review – identifying useful features – often requires an amount of research into how the technology works. It can be useful to develop a technology overview document based on your findings. This allows the work of one or two individuals to be spread in a meaningful way to other people in the organization. A firm understanding of the fundamental principles behind a technology can be very useful when performing a technology assessment.
Read more about performing a market review.
Phase 2: Testing
Many organizations do not move straight from the market review to the purchasing step, without any sort of hands on assessment. Often, they will find a device that meets their needs on paper, and purchase accordingly. There may be additional research – calls to peers or others experienced in the field – but devices will not always be brought in for comparison. This can be an expensive mistake.
The expense that may be incurred by failing to evaluate devices in a balanced, side-by-side test may not be immediately clear. To provide some context, consider the model of defect-discovery used within manufacturing processes that is often referred to as the “1-10-100 Rule.”
Preventing a mistake by properly planning and evaluating technology is relatively cheap. This cost increases by an order of magnitude if an organization has to correct the mistake. In the scenario of technology assessment, the initial cost is incurred by getting the devices in-house for testing. Not correcting the problem (implementing a technology without having first tested it to make sure that it will meet your needs) may cost yet another order of magnitude – more to fix – if fixing the problem is even a possibility.
This failure may result in provider or patient dissatisfaction, program failure or delay, increased programmatic costs, or, in extreme cases, potential harm to a patient. Acquiring testing devices can be incredibly important step in the course of evaluating the appropriateness of technology for an organization’s program. The following activities are typically associated with the testing process.
Step 3: Procure the Devices
If your organization has completed the work associated with performing a needs assessment and market review, deciding which devices to order for testing will likely be significantly easier. Compare the features that each product provides with the needs of your organization, and create a list of devices that may be successful in your program. Discuss this list with your interdisciplinary team to determine which devices should be procured for assessment.
At this point, a healthy relationship with vendors may be useful, as they can possibly negotiate deals on pricing of equipment, loaning devices for assessment purposes, or providing assistance while testing their products. Select a vendor or set of vendors who will meet your needs, and try to get all of the devices in for testing at the same time.
Step 4: Plan the Tests
Create a test plan, with supporting testing guides and documentation, before beginning any formal testing of the devices. Tests should include a combination of functional and mechanical criteria. Determine if any of the information gathered in the needs assessment can guide the testing process. Gather a team that can assist in testing the devices; an interdisciplinary group that represents technical and clinical interests is very useful at this point. At this early stage of testing, it may be beneficial to keep the team small, as coordinating schedules, performing tests, and discussing results can be easier with a smaller group.
Discuss the testing plan with the testing team, and make sure that everyone understands the criteria and scoring system that will be used during the assessment. Establish a schedule and set of deliverables that are expected from the test.
Step 5: Test the Plan
The completion of the actual hands-on testing of devices can often be completed fairly rapidly if the earlier steps of the assessment process have been completed. By having the testing planned in advance, a team that understands what is expected of the testing process, and all of the devices ordered and on-hand at the time of the assessment, it is relatively easy to finish the evaluation.
TTAC would like to emphasize the importance of testing all of the devices in as timely a manner as possible. Testing the devices together helps ensure that they are evaluated under similar circumstances, with a consistent process. Spreading the testing out over weeks or months makes it much more difficult to assure that the devices have been assessed equally. As the testing is performed, document the results of each test using the agreed-upon metrics from the test planning process.
Result Evaluation
Evaluation of testing results may begin in the actual testing process itself. Your testing process may allow testers to discuss their results in the process of assessing the devices. In contrast, some testing processes may call for results to be gathered individually, with the assessments only being discussed at the completion of the evaluation process. Either method for discussing the assessment results can be effective and the important outcome, regardless of process, is a mutual understanding of the results and a general agreement as to the performance of each device.
Additional conversation may need to take place as the results are reviewed. Significant differences in scores may need to be discussed to see why they occurred. These differences are more likely to occur on more subjective tests, as personal opinions and preferences will impact the results. Determine if it is most appropriate to average the scores, debate the differences until an agreement is reached, or if the test should be run again.
After looking at the numbers and discussing each device’s performance, it is likely that a subset of devices will be scored higher than the others. These devices can be put onto a “shortlist” of devices that may be looked at in greater detail.
Secondary Assessment
The shortlist that is created in the result evaluation may be subjected to additional assessment. This may be as simple as doing a final comparison against the initial needs assessment, or may go on to include hands-on testing with the clinicians who will use the device and technologists who will support the devices. The ultimate goal is a recommendation of a device or set of devices that will meet organizational needs for this technology.
Phase 3 Deployment and Support
The third phase of a technology assessment focuses on making your final selection, purchasing, deploying, and supporting that decision. This phase is both the completion and the continuation of all the assessment work described in this process so far. Special care needs to be taken to document choices that have been made, and to prepare for the operational realities of supporting a telemedicine technology.
Step 6. Select Device
With the hard work of testing, documenting, and device ranking completed, we are ready to make a selection and implement that decision. Key goals in this phase are to make a selection, communicate that decision, and clear documentation of our reasoning and results.
Testing Review
Those involved in the testing process may lose sight of the work that went into producing their final test results. Creating a document that provides a high-level review of the testing process can improve buy-in of individuals vested in the purchase of the equipment, and can explain why particular recommendations are being made. This can be especially important if any stakeholders not involved in the hands-on testing have a strong preference for a device that was not recommended at the end of the testing process. This document can also serve as a key resource in later technology evaluations, as it can quickly get personnel new to device evaluation up to speed on the general process fairly quickly.
This document will likely be easier to produce if the prior steps in the technology assessment process have been well documented- as much of the source material will already exist in those documents.
Information Sharing
The information gathered up to this point may be rather extensive. It is important to make a plan to disseminate relevant decisions and next steps to those who will be impacted by this work of the assessment team. Be prepared to inform clinicians and technicians about the decisions that have been made, including information about which products have been selected for implementation. Allow for feedback from interested parties, and be prepared to answer questions about the process that was followed, the impact of the decisions, and the plan for moving forward.
Read more about the decision making process.
Step 7: Deploy
Selecting a technology is only the start of a much larger process that involves purchasing and deploying the devices, as well as supporting deployed devices through their lifecycle.
Purchasing
Much of the work that happened earlier in the process will be used to drive many of the purchasing decisions. Having documented work done around planning and needs assessment will prove useful at this point. Your organization will have to decide how devices are bought. For example, will each receiving site be responsible for procuring devices on their own, or will a single organization purchase in bulk and distribute them to the end sites? Will a single vendor be used? Healthy relationships with vendors will prove useful when weighing out different options.
Deploying
Many of the deployment decisions will be driven by purchasing decisions. Deployment information will need to be documented and shared with relevant parties, including how and when devices will be sent to sites, how they will be configured, and how they will interoperate with existing infrastructure. Clear communication and thorough documentation of the deployment plan will prove to be critical to coordinating the deployment process.
Read more about purchasing, deployment, support and training.
Conclusion
Implementing a new or expanded telehealth program is a complex issue, especially when it requires a technology that is unfamiliar to your organization. That said, an effective technology assessment can be done by nearly anyone willing to take the time to walk through the process. It can be tempting to forgo actually obtaining and evaluating devices, as it requires a set of skills that often are in short supply in many healthcare organizations. However, the process of evaluation can help ensure that telehealth technologies are implemented in a way that meets the needs of a program. Working through the process in a methodological way will ultimately help an organization make more cost-effective purchasing decisions with a higher degree of confidence in final product selection.
The remainder of this toolkit, which can be accessed by the menu to the right, provides a more in-depth analysis of the work that goes into performing a technology assessment, with example documentation and evaluation guides to help in the evaluation process.
Additionally, the menu to the right includes links to toolkits for specific technologies that have been evaluated by the TTAC. These toolkits may provide examples of how technology assessment may be completed.