This study tackles a common, yet underrated problem in remote sensing image analysis: the fact that human interpretation is highly variable among different operators. Despite current technological advancements, human perception and interpretation are still vital components of the map-making process. Consequently, human errors can considerably bias both mapping and modelling results. In our study we present a web-based tool to quantify operator variability and to identify the human and external factors affecting this variability. Human operators were given a series of images and were asked to hand-digitize different point, line, and polygon objects. The quantification of performance variability was achieved using both thematic and positional accuracy measures. Subsequently, a series of questions related to demographics, experience, and personality were asked, and the answers were also quantified. Correlation and regression analysis was then used to explain the variability in operator performance. From our study we conclude that 1) humans were seldom perfect in visual interpretation, 2) some geographic objects were more complex to accurately digitize than others, 3) there was a high degree of variability among image interpreters when hand-digitizing the same objects, and 4) operator performance was mainly determined by demographic, non-cognitive and cognitive personality factors, whereas external and technical factors influenced operator performance to a lesser extent. Finally, the results also indicated a gradual decline in performance over time, mimicking classical mental fatigue effects.