This is true, and decisions don’t have necessarily have to be made by the general public. But they should still be kept “in the loop”. From the classic “Producing Open Source Software” by one of the fathers of open source, Karl Fogel:
Even after you’ve taken the project public, you and the other founders will often find yourselves wanting to settle difficult questions by private communications among an inner circle. This is especially true in the early days of the project, when there are so many important decisions to make, and, usually, few volunteers qualified to make them. All the obvious disadvantages of public list discussions will loom palpably in front of you: the delay inherent in email conversations, the need to leave sufficient time for consensus to form, the hassle of dealing with naive volunteers who think they understand all the issues but actually don’t (every project has these; sometimes they’re next year’s star contributors, sometimes they stay naive forever), the person who can’t understand why you only want to solve problem X when it’s obviously a subset of larger problem Y, and so on. The temptation to make decisions behind closed doors and present them as faits accomplis, or at least as the firm recommendations of a united and influential voting block, will be great indeed.
Don’t do it.
Of course, the question is where to draw the line between trivial decisions and important decisions. Nevertheless, in open source projects, it’s best if both types changes can be discussed publicly, regardless of who’s making the final decision (and how quickly).