This is really, immensely, cute and funny and all but…he was mostly talking to the woman, Dr. Stapleton and not John. He pointed right at her, addressing her only, which is why she is the one who replies and John looks between them. Sherlock wanted the annoying female to leave and John was all too happy to go with her. John knows what Sherlock is like in his Mind Palace, as he’s been around to witness it.