Technology Assessment 101 - Process Overview
Implementing a telehealth system requires the collaboration of an interdisciplinary team, as telehealth covers issues that are equal parts clinical and technical, operations and systems. Whether establishing a new program or expanding an existing program to include additional clinical specialties, a variety of people need to be pulled into the planning and implementation process. Many of the areas of expertise may exist within an organization, with clinicians, administrators, case managers, and networking staff able to provide input, yet one significant role is missing from this lineup – the technical expert that can recommend appropriate technologies for implementation.
Organizations may attempt to compensate for the lack of a technical expert by asking clinicians to select the devices that they will use to meet their needs, or by asking IT staff to choose the equipment that they will eventually support. Interdepartmental teams may be built, vendors may be brought in, and a decision may eventually be made.
There are two likely outcomes of this slightly-haphazard process: organizations build up expertise over time and gradually develop a process that is used for selecting equipment, or else they make purchasing decisions that eventually fail to meet their needs. Based upon feedback from various telehealth programs, organizations rarely have the time or staffing to reach the first outcome, and often know of at least one other program that has been bitten by the fallout from a misplaced technology purchasing decision.
At its most benign, a failed technology selection may result in a product that works in a telehealth environment while delivering a user experience that isn’t as polished as might exist with other products. Perhaps the device is a little hard to use, or perhaps the data being gathered or transmitted is simply good enough, rather than great. These situations can help educate an organization as to what does and does not work well for them.
At the more dangerous side of potential outcomes are technology selections that result in failed pilots, delayed programs, system incompatibilities, clinical user dissatisfaction, and at their worst, harm to patients. Organizations clearly want to avoid these negative outcomes, but many don’t quite know how to do so.
The goal of this toolkit is to provide organizations with a basic understanding of the work that should go into a technology assessment so that a consistent and repetitive process can be put into place when selecting technologies. The processes and procedures recommended here will hopefully assist in the decision making process, helping an organization’s interdepartmental teams to make choices that meet needs, extend the reach of their programs, and produce results that work well into the future.
That said, less-than-ideal purchases may still be made. By following a standard guide, however, the impact may be reduced, and an organization may be able to establish where and why the decision was made and how the process can be improved. We at TTAC strongly recommend that organizations refine their policies and procedures to meet their needs as they gain more experience in technology assessment.
There are several steps that should be performed in the course of technology assessment. The exact process may have slight variation due to differences in each technology, organizational competencies, and available resources. Generally, though, a thorough technology assessment should include the following steps:
- Information Gathering
- Needs Assessment
- Market Review
- Device Ordering
- Test Planning
- Test Execution
- Result Evaluation
- Secondary Testing
- Decision Making
- Testing Review
- Information Sharing
The following sections will provide a high-level description of the above steps.
Information Gathering Overview
The decision to perform a technology assessment typically stems from a separate decision to implement a telehealth system, or to expand services in a system that already exists. It is critical to pull together all of the available information prior to beginning a technology assessment, as this data will inform future decisions made by your organization.
As mentioned above, the choice to perform a technology assessment stems from a need to find a product that will best support a new or expanded telehealth installation. It is useful to take time prior to beginning the actual assessment of the technology to perform an assessment of your organization’s needs. What clinical needs will the device or devices have to meet? What patient population are we going to serve with this deployment? What workflows will need to be changed to implement the technology successfully? What buy-in do you have from clinicians, administrators, and technical support?
Look beyond the basic need, such as “we need to do teledermatology,” to find more meaningful organizational needs. A SWOT (Strength, Weakness, Opportunity, Threat) assessment may help in this process. Additionally, performing a SWOT assessment can help you determine which resources you have available within your organization that will assist in reviewing the technology, implementing the system, and supporting it into the future.
Once your organization has a firm grasp of its needs and resources, it is important to spend some time looking at what the market can offer. Some organizations will struggle with this particular step, as it can be hard to understand what differentiates one product from another, and what features are truly important to your program. Dedicating time to learn about the market can help your organization make informed purchasing decisions later in the process.
Gathering information can be challenging with many technologies used in telehealth. Each manufacturer will often use different terms to describe similar functionality, making it hard to establish any level of feature parity between devices. Additionally, some devices used in telehealth may come from a consumer-oriented market, resulting in discussion of features that are not pertinent to delivering telehealth. Discussions with vendors and manufacturers should be balanced with independent research into the technologies, so as to avoid some of the bias that occurs when talking to people with a vested interest in selling equipment.
A market review should include a comparison of features between devices, as well as an identification of which features will meet your needs. The output of the feature comparison should be a document that lists the potential devices on the market, including a tabularized comparison of all of the devices.
The second aspect of a market review – identifying useful features – often requires an amount of research into how the technology works. It can be useful to develop a technology overview document based on your findings. This allows the work of one or two individuals to be spread in a meaningful way to other people in the organization. A firm understanding of the fundamental principles behind a technology can be very useful when performing a technology assessment.
Many organizations do not move beyond the market review process of a technology assessment; they will find a device that meets their needs on paper, and purchase accordingly. There may be additional research – calls to peers or others experienced in the field – but devices will not always be brought in for comparison. This can be an expensive mistake.
The expense that may be incurred by failing to evaluate devices in a balanced, side-by-side test may not be immediately clear. To provide some context, consider the model of defect-discovery used within manufacturing processes that is often referred to as the “1-10-100 Rule.”
Preventing a mistake by properly planning and evaluating technology is relatively cheap. The cost increases by an order of magnitude if an organization has to correct the mistake – in the scenario of technology assessment, this is the cost incurred by getting the devices in-house for testing. Not correcting the problem (implementing a technology without having first tested it to make sure that it will meet your needs) may cost yet another order of magnitude more to fix, if fixing the problem is even a possibility.
This failure may result in provider or patient dissatisfaction, program failure or delay, or, in extreme cases, potential harm to a patient. Testing devices can be incredibly important in the course of evaluating the appropriateness of technology for an organization’s program. The following activities are typically associated with the testing process.
If your organization has completed the work associated with performing a needs assessment and market review, deciding which devices to order for testing is made significantly easier. Comparing the features that each product provides with the needs of your organization, create a list of devices that may be successful in your program. The list may be expanded or shortened based upon additional discussion while planning which devices will be ordered.
At this point, a healthy relationship with vendors may be useful, as they can possibly negotiate deals on pricing of equipment, loaning devices for assessment purposes, or providing assistance while testing their products. Select a vendor or set of vendors who will meet your needs, and try to get all of the devices in for testing at the same time.
Create a test plan, with supporting testing guides and documentation, before beginning any formal testing of the devices. Tests should include a combination of functional and mechanical criteria. Determine if any of the information gathered in the needs assessment can guide the testing process. Gather a team that can assist in testing the devices; an interdisciplinary group that represents technical and clinical interests is very useful at this point. At this early stage of testing, it may be beneficial to keep the team small, as coordinating schedules, performing tests, and discussing results can be easier with a smaller group.
Discuss the testing plan with the testing team, and make sure that everyone understands the criteria and scoring system that will be used during the assessment. Establish a schedule and set of deliverables that are expected from the test.
The completion of the actual hands-on testing of devices can often be completed fairly rapidly if the earlier steps of the assessment process have been completed. By having the testing planned in advance, a team that understands what is expected of the testing process, and all of the devices ordered and on-hand at the time of the assessment, it is relatively easy to finish the evaluation.
It is important to reiterate the importance of testing all of the devices in as timely a manner as possible. Testing the devices together helps ensure that they are evaluated under similar circumstances, with a consistent process. Spreading the testing out over weeks or months may make it so that devices are not assessed equally. As the testing is completed, document the results of the tests using the agreed-upon metrics from the test planning process.
Evaluating the results may begin in the actual testing process itself, as your testing process may allow for testers to discuss their results while assessing the devices. Some testing processes may call for results to be gathered independently, with the assessments being discussed at the completion of the evaluation process. Each method for discussing the assessment results has costs and benefits; the important outcome, regardless of process, is a mutual understanding of the results and a general agreement as to the performance of each device.
Additional conversation may need to take place as the results are reviewed. Significant differences in scores may need to be discussed to see why they occurred. These differences are more likely to occur on more subjective tests, as personal opinions and preferences will impact the results. Determine if it is most appropriate to average the scores, debate the differences until an agreement is reached, or if the test should be run again.
After looking at the numbers and discussing each device’s performance, it is likely that a subset of devices will be scored higher than the others. These devices can be put onto a “shortlist” of devices that may be looked at in greater detail.
The shortlist that is created in the result evaluation may be subjected to additional assessment. This may be as simple as doing a final comparison against the initial needs assessment, or may go on to include hands-on testing with the clinicians who will use the device and technologists who will support the devices. The ultimate goal is a recommendation of a device or set of devices that will meet organizational needs for this technology.
Decision Making Overview
The results of the testing process are often understood very well by those who performed the evaluation, but may not be immediately understood by other stakeholders. Additional steps should be taken to ensure that the data is clearly disseminated to key people involved in the decision-making, purchasing, and implementation processes.
Those involved in the testing process may lose sight of the work that went into producing their final test results. Creating a document that provides a high-level review of the testing process can improve buy-in of individuals vested in the purchase of the equipment, and can explain why particular recommendations are being made. This can be especially important if any stakeholders not involved in the hands-on testing have a strong preference for a device that was not recommended at the end of the testing process. This document can also serve as a key resource in later technology evaluations, as it can quickly get personnel new to device evaluation up to speed on the general process fairly quickly.
This document will likely be easier to produce if the prior steps in the technology assessment process have been well-documented; much of the source material will already exist in the other documents.
The information gathered up to this point may be rather extensive. It is important to make a plan to disseminate relevant decisions and next steps to those who will be impacted by this work of the assessment team. Be prepared to inform clinicians and technicians about the decisions that have been made, including information about which products have been selected for implementation. Allow for feedback from interested parties, and be prepared to answer questions about the process that was followed, the impact of the decisions, and the plan for moving forward.
Purchasing and Support Overview
Selecting a technology is only the start of a much larger process that involves purchasing and deploying the devices, then supporting them throughout their lifecycle.
Much of the work that happened earlier in the process will be used to drive many of the purchasing decisions. Having documented work done around planning and needs assessment will prove useful at this point. Your organization will have to decide how devices are bought – will each receiving site be responsible for procuring devices on their own, or will a single organization purchase in bulk and distribute them to the end sites? Will a single vendor be used? Healthy relationships with vendors will prove useful when weighing out different options.
Many of the deployment decisions will be driven by purchasing decisions. These decisions will need to be documented and shared with relevant parties, including how and when devices will be sent to sites, how they will be configured, and how they will interoperate with existing infrastructure. Clear communication and thorough documentation of the deployment plan will prove to be critical to coordinating the deployment process.
A clear and complete support plan should be in place before the devices are deployed to the receiving sites. It is important to think through how your organization will handle problems that occur during installation and deployment, as well as how warranty issues will be resolved, how device failures will be addressed, how user-based problems will be handled, and how equipment will be maintained into the future. Documenting these issues and discussing them with end users will help an organization when inevitable support issues occur.
As with support, training is an ongoing issue after the initial device deployment occurs. It is important to budget for training costs, and to have a formal plan in place to ensure that users are kept current on their understanding of the technology being deployed. Who will be responsible for educating both existing and new employees? Will standard curriculum be developed? Does your organization have a training program in place that can be expanded to include telehealth education?
Implementing a new or expanded telehealth program is a complex issue, especially when it requires a technology that is unfamiliar to your organization. It can be tempting to forgo the process of evaluating devices, as it requires a set of procedures that are do not typically exist in healthcare organizations; however, the process of evaluation can help ensure that a telehealth technology is implemented in a way that meets the needs of a program. Working through the process in a methodological way will ultimately help an organization make more cost-effective purchasing decisions with a higher degree of confidence in the appropriateness of the final product selection.
The remainder of this toolkit, which can be accessed by the menu to the right, provides a more in-depth analysis of the work that goes into performing a technology assessment, with example worksheets and evaluation guides to help in the evaluation process.
Additionally, the menu to the right includes links to toolkits for specific technologies that have been evaluated by the TTAC. These toolkits may provide examples of how technology assessment may be completed.