App stores provide access to millions of different programs that users can download on their computers. Developers can also make their programs available for download on their websites and host the program files either directly on their website or on third-party platforms, such as mirrors. In the latter case, as users download the software without any vetting from the developers, they should take the necessary precautions to ensure that it is authentic. One way to accomplish this is to check that the published file’s integrity verification code – the checksum – matches that (if provided) of the downloaded file. To date, however, there is little evidence to suggest that such process is effective. Even worse, very few usability studies about it exist.
In this paper, we provide the first comprehensive study that assesses the usability and effectiveness of the manual checksum verification process. First, by means of an in-situ experiment with 40 participants and eye-tracking technology, we show that the process is cumbersome and error-prone. Second, after a 4-month long in-the-wild experiment with 134 participants, we demonstrate how our proposed solution – a Chrome extension that verifies checksums automatically – significantly reduces human errors, improves coverage, and has only limited impact on usability. It also confirms that, sadly, only a tiny minority of websites that link to executable files in our sample provide checksums (0.01%), which is a strong call to action for web standards bodies, service providers and content creators to increase the use of file integrity verification on their properties.