Security tools for macOS focus on visibility and control: outbound firewalls, persistence monitors, camera and microphone alerts and code signature checks. They add a layer on top of Gatekeeper and XProtect, mostly by showing what software is doing so you can decide whether to allow it.
Several of the best known come from Objective-See and are free and open source: the LuLu firewall, BlockBlock for monitoring persistent installs, KnockKnock for listing persistent software, OverSight for camera and mic alerts, and What's Your Sign? for checking signatures. Santa is a binary allowlisting tool aimed at managed fleets. Expect approval prompts for system extensions.