Generally, no. It is a common misconception that it takes more energy to heat up a cold house than it does to keep a house warm all the time. Turning down the thermostat will almost always save energy, as will turning up the air conditioner temperature setting. Heat moves from hot to cold, and the rate of heat transfer increases with greater temperature differences between inside and out. Smaller temperature differences between your house and outside generally means you'll lose less heat (or air conditioning) from inside the house to the outside. Therefore, you will save energy by only heating or cooling as much as necessary, for the occupants and time of day. The one exception is if you use much less efficient heat (or cooling) to get back to your desired temperature - for instance if you have a heat pump and the "aux" light comes on. Some programmable and many smart thermostats are able to avoid this problem using "heat pump recovery" and you can set them back without worry, even if you have a heat pump.
If your concern is comfort, or having to wait for your house to heat up or cool down after you get home, invest in an ENERGY STAR certified smart thermostat that can begin to heat or cool your home early, so the temperature is just right when you arrive. Through proper set back and set up, you can save about $180* every year in energy costs for a typical, single-family home, compared to keeping your home comfortable all the time. *The $180 savings assumes an 8 hour daytime setback and a 10 hour nighttime setback of 8 degrees F in winter and 4 degrees F in summer.