What's here called the His Majesty's Cottage became the Royal Lodge, the residence at present of Prince Andrew. It's by no means the same building. King William IV had all but the conservatory demolished after his brother King George IV* died. Why? Good question. I wondered the same about King George IV’s demolishing Carlton House.
Ackermann’s Repository for January 1823.
*More about this controversial monarch here, here, and here.