The X sign, erected Friday atop the building in downtown San Francisco, was removed after complaints from residents and a city citation.