We know from the example of Treebeard that the Ents are morally ambivalent. No argument about the good of the world that Merry and Pippin make sways him; the only reason the Ents march on Isengard is that their cosy little forest was being disrupted. Yes, they hate Orcs, because Orcs tear down trees - but remember this, from the very creation of the Ents:
"Eru is bountiful," [Yavanna] said. "Let thy children beware! For there shall walk a power in the forests whose wrath they will arouse at their peril."
"Nevertheless they will have need of wood," said Aule, and he went on with his smith-work.
Dwarves, Men - even Elves cut living wood for their own purposes. The wrath of the Ents was aimed at all of them, and for good reason: the trees being cut down were their kin, living things which had at least the potential to become intelligent, speaking beings.
Treebeard describes Ents as 'a secret in the heart of the forest', but what he really means is 'a hidden terror'. Remember the reaction of the Dunlendings to the forest outside the Hornburg - fear, panic, horror. That isn't just an appearing forest - it's a children's ghost story come to life. The Trees Which Eat The Careless Axeman would have been a figure of fear across the old world, since before the arrival of the Numenoreans.
Yes, before Numenoreans settled in Gondor and Rohan. We know that the Ents and the Entwives were not all that close; Treebeard tells us this:
"After the Darkness was overthrown [at the end of the First Age] the land of the Entwives blossomed richly, and their fields were full of corn. Many men learned the crafts of the Entwives and honored them greatly; but we were only legend to them, a secret in the heart of the forest."
So who were the Entwives teaching, east of Anduin, right up against the Emyn Muil, before the rise of Gondor? The inhabitants were the Men of Darkness, those who never made the journey to Beleriand; the ancestors of the Dunlendings, in other words.
Consider the timeline of the Second Age:
In 1600, Sauron forges the One Ring in Mordor, and raises Barad-dur.
In 1693, Sauron wages war on Eregion, west of the Misty Mountains.
In 1699, all of Eriador is under Sauron's sway.
In 1700, Numenorean ships drive Sauron out of Eriador and away from the coasts.
In 1800, Numenor begins to settle the coasts and some of the lands west of the Misty Mountains.
In 3262, Sauron is taken prisoner by the Numenoreans.
... and for all that time, the Entwives are peacefully tending their gardens, right on the path between Sauron's domain and the West.
There is only one reasonable conclusion: the Entwives sided with Sauron. The men who 'learned the crafts of the Entwives' and 'honored them greatly' were true Men of Darkness - worshippers of Sauron the Abhorred, Dark Lord of Mordor. That's who was living in their area; there can be no doubt.
What did Sauron get out of this arrangement? Simple - food for his armies. The heart of his territory was a bleak wasteland, and even Nurn doesn't seem to be that fertile. The gardens of the Entwives were a ridiculous bounty - particularly since they trained the men who worshipped him to the same standard. From Sauron's point of view, the Entwives and their students were the best thing that could ever happen to him.
And what did the Entwives get out of it? Take a look at their description, as presented by Treebeard, who once loved them:
They did not desire to speak with these things; but they wished them to hear and obey what was said to them. The Entwives ordered them to grow according to their wishes, and bear leaf and fruit to their liking; for the Entwives desired order, and plenty, and peace (by which they meant that things should remain where they had set them).
Order. Obedience. 'Peace' with themselves in the position of power. No complaining from their subjects. The desires of the Entwives are exactly the same as the desires of Sauron - dominion.
So what happened to the Entwives? Treebeard tells us: the War of the Last Alliance swept over their lands, and their gardens were burned, becoming the Brown Lands. But the War of the Last Alliance was an attack by the Numenoreans on Sauron, not the other way round. Sauron attacked Gondor - taking Minas Ithil, and assailing Osgiliath and Minas Anor. But the Brown Lands? They lie on the route to Mordor from the elven domains west of the Misty Mountains. The only force to pass through the gardens of the Entwives under arms was the army of Elves and Men. The 'breadbasket of Sauron' was burned by the Last Alliance.