Burn probability maps produced by Monte Carlo methods involve repeated simulations of fire ignition and spread across a study area landscape to identify locations that burn more frequently than others. These maps have achieved broad acceptance for research investigations and strategic fire management planning. In this study, we investigated correspondence between burn probability heat maps and burned areas observed in subsequent years for five study areas in Alberta, Canada. Observations of burned areas included 138 fires that collectively burned 543 049 ha. Distributions of burn probability values within burned areas were not heavily skewed towards the high-end of the range; however, median burn probability was significantly lower in unburned areas compared with burned areas for three of five study areas. Accuracy of burn probability maps was dependent on map design choices and subjective performance thresholds. When continuous burn probability values were mapped with a stretched symbology, most observed burned areas (>70%) were located in areas within the lower-half of the burn probability range where fires were considered the least likely. In contrast, when burn probabilities were mapped and evaluated with a 50th percentile performance threshold, most observed burned areas (75–80%) occurred in half of the study area where burn probability values exceeded the median. Map accuracy declined linearly as this performance threshold was increased from 50th to 90th percentile. Choice of classification method for mapping burn probabilities altered the appearance of the map and corresponding map accuracy. Compared with Jenks natural breaks, equal intervals, or defined intervals, a quantile classification was the only method that resulted in burned areas falling preferentially in locations mapped in the highest burn probability classes.