The Sunday Tribune article titled “Software hiccups plague student testing” brought on a bout of déjà vu.
As a former IT director for the Utah State Board of Education, I observed or was a member of selection committees that reviewed bids made in response to requests for proposals. These bids ranged from software used to collect data for state and federal reporting to systems for statewide testing programs. It is not surprising to me that the USBE is having difficulties with yet another vendor.
While there are always risks in writing any RFP and selecting a vendor deemed to have presented the best overall bid, there are certain factors that often result in a poor choice. Perhaps the most problematic are state purchasing rules that often give too much weight to the lowest bidder. In many RFPs cost accounts for over 40% of the total evaluation points.
Other factors involve the membership of the selection committees that review the bids submitted in response to the RFP and award contracts. When forming these committees, the USBE often overlooks two important groups of individuals. First, are the technology experts. These are individuals with in depth experience in purchasing, building, using and maintaining software systems. They are often able to identify hype, ambiguous jargon and subtle misrepresentation that are often found in bids.
As an added benefit, these committee members can contact technology staffs in other states and school districts. These contacts often reveal software characteristics that are not revealed through specifications, presentations, demonstrations or endorsements. Sometimes this is the only way to get reliable information about the software system’s capacity.
Second, the actual users of the software — including teachers, school support staff and the students — are often overlooked. If these individuals are not present when vendors demonstrate their software and are not able to try it themselves, then deficiencies, unnecessary features, performance issues and clumsy processes may be overlooked.
When these groups of users are not included, the committee will often miss problems with the way those users must interact with the software. What may look like an intuitive and efficient interface when the vendor does a demonstration is not always so understandable or easy for someone who has to sit down and use it in a school environment. Therefore, it is imperative that these users tryout the software in a school setting, and give feedback before the committee awards any contracts. In addition, this level of hands-on participation results in a higher level of confidence in all stakeholders; especially those at the school level.
Finally, committee members should be aware of any political forces that could influence the selection. For example, this could be the case with SB112 passed by the 2019 Utah Legislature. That bill appropriated $17.2 million to replace the USBE’s nationally recognized longitudinal data system.
Too often, a vendor can convince a legislator they have some unique product that will solve all a department’s software problems. Such a legislator may even try to influence a selection committee by expressing preference for one vendor, or by getting an individual who favors that vendor placed on the committee.
These types of behaviors did come into play during a large USBE software procurement in the late-2000s. They almost resulted in a contract being awarded to an underqualified vendor.
John Brandt
John Brandt, Ph.D., South Jordan, is a retired information technology director. During his career with the Utah State Board of Education, he served on numerous state and national education technology committees.