This document is intended to be used as a step by step guide through the Telehealth technology assessment process. While this is related to the TTAC patient exam camera toolkit assessment, parts of this can be adapted to any Telehealth technology assessment. This document takes the reader from start to finish thorough the process that the TTAC team used to assess patient exam cameras. It starts with a guide to help determine the technology need and walks the reader through exactly how to assess and test patient exam cameras.
Determine the minimum requirements
Establishing a base set of requirements is critical when starting any technology assessment. Determining a base set of requirements for the patient exam camera may look a bit different than assessing minimum requirements for other Telehealth technologies. There are really few industry recognized options available on the market to choose from and they all have fairly comparable functionality sets. Patient exam camera options are also limited by the fact that existing VTC units have very limited input options. With the patient exam camera, you may find yourself focusing more on determining the programmatic needs that the patient exam camera needs to fit. The patient exam camera is most times an adjunct to an existing Telehealth technology, whether it is an asynchronous or synchronous platform, which allows for some potential flexibility in product selection. Therefore, you may also find yourself determining if there are other technologies on the market, aside from industry recognized “patient exam cameras,” that might be able to fill your need while still being technically sufficient. The TTAC focused on assessing a wide variety of potential patient exam camera models, in order to hopefully represent some additional technologically sufficient less recognized options.
Define the users
Knowing the users that will be utilizing the patient exam camera is an important first step in creating your programs minimum requirements. Knowing the users will help you determine the acceptable level of complexity that you can comfortably introduce with your selection. Questions to ask include:
- Which specialties will use the patient exam camera?
- What is the level of expertise of the users?
- How much time do the clinical users have to devote to an encounter utilizing the patient exam camera?
- On average, how many times per day will the patient exam camera be used?
- What other technologies do the users use on a daily basis?
Define the required functionality
Various user groups will need different functionality and performance from the patient exam camera. If the cameras will be shared by multiple disciplines make sure that all needs are represented. Being able to separate out the clinically useful features and functionality from the clinically irrelevant features and functionality will also aide in selecting the most appropriate technology. Being acutely aware of the required functionality for the clinical need, will possibly open your selection options up beyond the industry recognized “patient exam cameras” currently on the market.
- What is the clinical need for using the patient exam camera?
- What types of surfaces will the patient exam camera be imaging?
- How important is color accuracy?
- Are there wet surfaces?
- At what distances will the patient exam cameras be used at?
- Are there needs to capture up close (macro) shots?
- Is there a need to utilize zoom functionality in the patient exam camera?
- What platform will the patient exam camera be used with: asynchronous, synchronous, or both?
- Is there a need to store images?
- Is there a need to send image over a network connection?
- Is there a need to integrate the data obtained with the patient exam camera into an EMR?
- What existing equipment will the patient exam cameras be used with?
- What types of cables, connections, and/or converters may be necessary to connect the patient exam camera to the existing equipment?
- Do the users need to be able to hold the camera or will they be using the camera mounted on a tripod?
- Do the users need a light-weight patient exam camera?
- Does the patient exam camera need to be used one handed?
- Do the users need to be able to cleanse the patient exam camera between patient uses?
- How many menu selections to obtain a feature or functionality are acceptable the users?
- Is there a high need for mostly automated feature sets?
- What environment will the patient exam camera be used in?
- What is the noise and lighting level naturally present in the intended use environment?
- Based on the clinical need for the patient exam camera, what level of external light will be necessary to obtain quality images?
Define the deployment
Establishing where the patient exam cameras will be deployed to will most likely be based on the required functionality and clinical need. The level of training and support the patient exam cameras will require will largely be based on the user assessment. All of this information will be helpful when budget and planning for the integration of patient exam cameras into your program. It is important to keep deployment and support in the forefront of your mind throughout the entire technology assessment process, because it affects all aspects of the assessment and will aide you in making the most appropriate technology decisions for your program. Refer to the “Deployment and Support” section of the toolkit for a more in depth explanation. Knowing your site equipment and support needs will help define your budget and could ultimately be a very important factor in your equipment selection. It is important to be open minded yet extremely mindful of your determined user population and required functionality as you move forward in the assessment process. Spending adequate time planning will save time and potential headache later in the assessment process.
Assessment
With clear ideas about your intended user population and potential equipment use, you are now ready to continue along the assessment continuum. You are ready to start actually evaluating the technology. You will start broadly with a complete market assessment based on your needed functionality, determine the units you will bring in for testing, gather all the information you can about the units you bring in to assess, come up with a testing plan based on the confirmed functionality of the equipment, actually test the patient exam cameras that you bring in, and then begin to make some decisions based on your needs and results obtained.
Market Review
With your minimum requirements in mind, start with a broad internet and manufacturer search for all possible technology types on the market. Initially use the market review to familiarize yourself with what is available on the market. After you have determined what is available, then it is time to start molding the availabilities against your minimum requirements, programmatic needs, and clinical needs to start to narrow down your options.
Being methodical about your searches and documenting the options will make your market review much easier, especially if you are working with others or will need to be able to show your results to others. The process of elimination can begin once you have compiled a list of patient exam camera options. Patient exam cameras that do not meet your minimum requirements should be removed from the list. You may need to get creative to determine a devices suspected functionality. Some useful resources are online product manuals and cut sheets, online reviews, manufacturer and reseller websites, and speaking directly with vendors. You may also find the “Resources & Standards” toolkit section helpful.
Your list of patient exam cameras should be manageable at this time. The actual number of cameras that you bring in for testing, and the amount of testing that you will be able to do, will depend entirely on the budget that you have set aside. Be very cautious about making large volume technology selections without first laying hands on the equipment and personally verifying the functionality.
Once you have made your selections, order test units or sample units and have delivered to begin the testing process.
Product Information Gathering
For the TTAC, the product information gathering phase is really the first stage in testing. We begin to familiarize ourselves with the technology as we record information on it. You will have inevitably already gathered some product information during your market review. Once you have the equipment in-house, you can gather more information about the products from their packaging and manuals. Gathering the information that will be important during testing is very useful, and should be in line with the minimum requirements that you have already established. The multiple formats of information can become overwhelming, which is why gathering product information into your own personal format can be very useful. See the “Product Information” section of this toolkit to see the information that TTAC focused on for the patient exam camera assessment, as well as the table we used to summarize the information we gathered on all models.
Test Planning
It is tempting to begin “testing” the equipment as soon as you receive it. However, planning your testing against your minimum requirements and your programmatic needs will make the labor intensive testing process more manageable in the end. Without a plan for testing, you run very distinct possibility that you will select a model because it is “cool” but turns out to be incredibly un-useful and/or inappropriate for your programs needs. Also, without a testing plan you will inevitably utilize too many resources (financial and personnel related) and possibly jeopardize other aspects of your budget. Keep in mind that the planning is just that, planning. It gives you a chance to document what you think you might do during testing. Oftentimes, what you plan on doing may change a bit once you get in the room with your team and have hands on the equipment. You may also test things and/or gather data that you later determine to not be useful. The more assessments you do, the better your team will get at planning.
Actual Testing
Testing can be a time and labor intensive process. Roughly sticking to your plan will help, but you need to expect to spend a full day or potentially multiple days with your team. Assigning different roles to your team members can also make testing run more smoothly.
Having someone that can document the aspects of testing is extremely helpful, if you don’t have the manpower to have someone documenting you might video tape or tape record the session for a refresher. The goal of documenting is to work towards a reproducible process as testing emerges again in the technology assessment cycle.
Reserve enough space to comfortably be it the same room with your team and all of the equipment that you purchased for testing/evaluation. Remember if you are going to be obtaining sample media clips or images from volunteers, you also need a secure location that can provide dignity and privacy. The TTAC team used a room that had a VTC unit, network connectivity, and a large enough table that they could lay out each patient exam camera with its associated cables and attachments. Having all the necessary equipment and personnel in one place will help the process run smoother.
It is helpful to introduce rigor into the testing process whenever you have the opportunity, it will add validity to you data and increase your overall confidence in your testing process. In the patient exam camera evaluation, the TTAC team introduced rigor in many ways. Most notably: we used one VTC codec, one continuous connection, we assessed each characteristic with the same patient exam camera order, and while collecting clips we utilized the same imaging guidelines.
Sample Media Acquisition
In order to compare media for color accuracy and image detail, the TTAC team gathered a set of clinical images with each patient exam camera utilizing the same set general set up with variation when it came to the camera connections. The images that we obtained were: back, oral, eye, scar, Vitiligo. The sample video that we captured was a framed face doing an abbreviated facial nerve exam. We attempted to frame the shots with as little variance as possible, utilizing measurement standards and standardized positioning for obtaining each image set.
We utilized an IMPERX VCE Pro Custom PCMCIA Frame Grabber, attached to the PCMCIA slot on a Dell Latitude D630 Laptop using the IMPERX VCE Pro Software Application to capture the images. All images were captured as JPEGS at full frame (640×480) and all videos were captured in AVI format, at 30 frames per second, at full frame (640×480). We documented the variances encountered with each camera here. Images and videos obtained can be viewed in the “Sample Media” section of the patient exam camera toolkit.
After collecting the images we input them into a customized application that allows for simultaneous image comparison. Each image was assessed with a 1-4 likert scale for color accuracy and image clarity. See the Image Evaluation table for 2010 TTAC data.
Functional testing
The TTAC team decided upon “Mechanical” and “Ease of use” functional testing elements to test each patient exam camera against. The elements you select should be based on your minimum requirements and be mindful of the clinical needs and user population.
For the TTAC functional testing we assembled the TTAC team in a large room with network access. We placed each patient exam camera with its associated cables in a line on a large table. We utilized a 50 inch Sharp TV, a Polycom HDX 7000 dialed into a secured meeting room, a RMX 2000 Polycom Bridge, and an S-Video cable plugged into the VCR input on the HDX 7000.
We utilized a table document to simultaneously document the assessment results as we assessed each element. We assed each element one at a time, and compared it among each patient exam camera in the same sequential order. We had rough definitions ahead of time for each element that we were assessing to guide our frame of reference. As we assed each element, we recorded ratings in our documentation. We used a 1-5 Likert scale, where five represented the best score. Please review the TTAC testing element results table to review our results.
Conclusions
Once you obtain all of your data from testing, you will again need to compare it to your minimum requirements. Some models of patient exam camera will easily be eliminated because you will find that they don’t meet some requirement or have poor quality or functionality. Overall, it will depend on what you are trying to prove with your assessment and testing. You can use likert based ratings as tiebreakers, but be mindful that not all elements are created equally and may deserve different weightings in your overall summing or averaging. It is good to have a wide range of data components from testing, but don’t allow too much data to sidetrack you from the important details. Be mindful of the whole testing process, and what it took to get each piece of equipment to do what you needed it to do. You may discover some unexpected or unanticipated results after your assessment.
You may have been using the technology assessment and testing process to see if using a new type/model of patient exam camera could meet your programmatic need. By the time you are to the point of testing being concluded, you will know if an alternative model will be technologically sufficient to meet your programmatic need.
A decision is not always clear, so you will most likely need to go back to your assessment results and make a decision regarding technology selection. It might be helpful to wait a short period of time that allows all of the details to sink in. Be careful not to allow too much time between decision making and testing, as to not forget testing details.