The setting you want is “allowed extensions”. Remove all extensions from this list, and the option to upload will disappear from the composer. Looks like the helper text for this setting could stand improving to make this more obvious.
Edit: hmmm I just realized this also disables avatar uploads. Maybe a more knowledgeable person can give a better answer. Or this is a feature request to add a setting to disable file uploads while allowing avatar uploads.