Latency or system response time (i.e., the delay between user input and system response) is a fundamental factor affecting human-computer interaction (HCI). If latency exceeds a critical threshold, user performance and experience get impaired. Therefore, several design guidelines giving recommendations on maximum latencies for an optimal user experience have been developed within the last five centuries. Concentrating on the lower boundary latencies, these guidelines are critically reviewed and contrasted with recent empirical findings. Results of the review reveal that latencies below 100 ms were seldom considered in guidelines so far even though smaller latencies have been shown to be perceivable to the user and impact user performance negatively. Thus, empirical evidence suggests a need for updated guidelines for designing latency in HCI.