Suggested time: 10 minutes
The following case study is an example of where algorithmic transparency can go wrong. Again, this case study is based on types of ADS that exist in the real-world, but is a fictional example.
In many US states, there are not enough food inspectors to evaluate every resteraunt each year. In an ideal world, government food and safety agencies would be able to hire additional staff to meet inspection demand. Instead, what they must do is prioritize where to spend inspectors based on where they think violations may occur, and how long it has been since a resteraunts las tinspection. Increasingly, government agencies have begun using algorithmic systems for this triage process.
At the beginning of each yearly quarter, managers at the food and safety agency run the algorithmic system processes all the data collected about the resteraunt and produces a list of resteraunts that can be reasonably visited by the inspectors on staff. The data includes informaiton about the resteraunt like its past violation history, the date since its last inspection, the type of cuisine it serves, and its location (sample data here: https://data.cityofnewyork.us/Health/DOHMH-New-York-City-Restaurant-Inspection-Results/43nn-pn8j). The resteraunts are rated on a score of 1 to 5, with 5 being those most in need of inspection. Note that the system makes sure to abide by all local and state laws in its final list generation (for example, every resteraunt must be inspected every 5 years, new resteraunts must be inspected within 1 year of opening, and resteraunts are subject to inspection no more than three times a year). The managers then assign an inspector to each resteraunt, and give the inspector the printout from the algorithm which contains the top 3 reasons contributing the algorithms score.
###
The algorithm’s output may not well understood by the inspectors. In the description, we are told that a score of “1 to 5” is output by the algorithm, but we are given no benchmark on the meaning of these scores. Why is a 3 different from a 4? A 4 from a 5?
Furthermore, generally speaking, telling inspectors only the top 3 attributes used by the algorithm is incomplete. First of all, the inspectors may not be aware of any other attributes used the by the algorithm. Second, what if the attributes outputted go against the subject area expertise of the inspector? In such a case, there is no way for them to support (or deny) the decision. Third, there may also be problems about the basic understanding of why the top 3 attributes are relevant: imagine if all the inspector was told was the location, cuisine type, and the last inspection date were the top 3 reasons an establishment is recommended. This leaves more questions open than answered.
The background also mentions no mechanisms for evaluating the overall performacne on the system. Is the system reducing the number of failed inspections, or the number of reported food-poisining instances in the state? There is also no mention of transparency related to fiarnesss. For example, are Domican or Mexican resteraunts being recommended more frequently for inspection as compared to American resteraunts with a similiar inspeciton history?
Additionally, nowhere at all in the background are the affected individulas (the resteraunts and their staff) even mentionedmention. Do those affected even know that their resteraunt is being evalauted by an algorithmic system? There is also certainly no mention of an opportunity for resteraunts to appeal the decision for an inspection before their case is assigned to an inspector. Inspections can be costly to a resteraunt (for example, in NYC, each time a resteraunt is selected they must pay the state $100), and so it is relevant they have at least some mechanism of recourse against the ADS.