Sometimes I find myself thinking "I'd know how I'd do this but I have no idea how to do it in Unity!" and this is always frustrating and the biggest downside of using a ready-made engine or - in this particular discussion - a 3rd party physics engine.
In our current project
DWARVES I've spend a lot of time on coding a C# library/framework (called HORDE for Horde Onslaught Realtime Dynamics Engine) which does things you'd typically expect an engine to handle for you. Most notably: Agent Behavior (AI), Pathfinding, Collision-Detection & Response, Simulation of Forces, Inertia etc, Scene-Management & Queries.
The simulation is essentiall 2D. There is a 3rd dimension (height) which plays a role but it's not fully supported. Think of a 2D world + heightmap. And everything is build from capsules and circles. There are no other shapes the engine can handle. Capsules can't even move. But as a player you'll hopefully never notice how simplistic the simulation models the world.
I decided on not using 3rd party physics because I would have used only a small subset of it anyway and felt comfortable about implementing that myself. Of course that's a time investment and I probably sacrifice some runtime performance too. But the good thing is that it's no problem to simulate more then just rigid-body physics. Or whatever other feature we need to get as close to our vision as possible. The framework I coded is very game-specific - it's not general purpose at all. But that's what is keeping it managable and understandable. Elegant even. And there are features in there that I wouldn't know how to do without having full control over the simulation.
I can't always justify to roll custom solutions (and that we're using Unity is testament to that) but I'm usually a happier coder when I'm working in an environment where there are no dependancies on 3rd party blackboxes to worry about.