The quality of the food we eat is more important than how much we eat, according to a new study that examined why people gain weight. And when it comes to the worst dietary culprits, potato chips, sugary drinks and meat top the list.

The study from Harvard School of Public Health researchers followed more than 120,000 people for 20 years. They found the average person gains 3.35 pounds over a four-year period, particularly if they ate regular servings of potato chips, potatoes, sugar-sweetened beverages, unprocessed red meats and processed meats.

Those who drank alcohol regularly were also found to generally gain weight, amounting to about 0.41 pounds in four years for every daily drink consumed. Ex-smokers gained an average of 5.17 pounds within four years of quitting, and people who slept more than eight or less than six hours per day gained 0.31 pounds over four years.

When it came to those who lost weight, good eating and exercise habits -- perhaps unsurprisingly -- made all the difference.

Regular exercise helped people lose an average of 1.76 pounds over four years, while the following foods were most common in participants who lost weight:

• Yogurt: -0.82 pounds over four years
• Nuts: -0.57 pounds over four years
• Fruits: -0.49 pounds over four years
• Whole grains: -0.37 pounds over four years
• Vegetables: -0.22 pounds over four years

The study's authors found that focusing on total calories, sugar or fat content didn't necessarily help people lose weight. Rather, those who eat unprocessed foods, and fewer sweets and starches are most likely to stay at healthy weights over the long term, or even shed a few pounds.

"The idea that there are no 'good' or 'bad' foods is a myth that needs to be debunked," said Harvard School of Public Health professor Frank Hu, a senior author of the paper, in a news release.

The findings, published in the New England Journal of Medicine, imply that a bit of positive change today can have major benefits over the long term.

"Small dietary and other lifestyle changes can together make a big difference -- for bad or good," said lead author Dariush Mozaffarian, of Harvard's epidemiology department, in a news release. "This makes it easy to gain weight unintentionally, but also demonstrates the tremendous opportunity for prevention. A handful of the right lifestyle changes will go a long way."