I know we live in a cold climate, but on occasion it does get blistering hot, especially in a car that's been sitting in the sun all day with the windows up.
The point is that there really is no benefit to removing it unless you have a dedicated track car.
It will ultimately detract from the value/saleability of the car.