You can disable an Actor, which disables everything related to that actor.
If you delete the category or tag the actor is associated with it will delete the Actor. Deleting an actor in isolation is not currently possible. I would recommend just disabling the Actor.
This may be possible in the future (medium to long term) if we add an “Application” actor for an entire Discourse. Federating everything on a Discourse through a single Actor will likely always be a niche use case.