You're better off with UFE and modding it to your needs  - which would only be fatalities, everything else you mentioned is either already included or coming in an update very soon.

Yes to both.  It's possible to do the latter to an extent now, but more options are available for it in 1.5 update (this was required for grapple reactions).  Check this thread for info, videos and demo.

EDIT: updated version reference

For Fatalities, I would suggest creating a new state the player can be in (an addition to stun and down states, you could call it dizzy or fatalityReady or something).  Then, when the round end check comes in, check if there's a winner and if we're the loser put us into this new state for a preset time.

To create the fatality move, just make a move that is only possible when the opponent is in this new state and the distance you want the move to be triggered from.  The move should have opponent override to control their reaction to the fatality too.

The question I can't answer is how to swap the model with the fatality version (the dismembered one) on the fly.  If that could be baked into an animation somehow, you could have it as part of the reaction move.  But I'm really not sure at all on that.

If you can solve the model swap, that should be all that's required. But I should mention, I haven't tried any of this .  Good luck!

EDIT: thought of the fatality model issue.

Yes, you'll need to edit code to add a new state.

Custom Editors need their files in an Editor folder.  If you are using MonoDevelop, you'll see that your project has at least two branches (I think that's what they're called):  Assembly-CSharp and Assembly-CSharp-Editor.  The editor one has all the editor files (expanding the branch, you can see the files in an Editor folder).  Anyway, all this means is to make a change to a UFE custom Editor Window, you'll need to change the appropriate editor file in the .\UFE\Editor folder.

But you won't need to edit the Move Editor file to get the new state to appear in Player Conditions.  Because if you just the add a new state to MoveInfo.cs as per code below, it will appear as part of the drop down.

public enum PossibleStates {
    Stand,
    Crouch,
    StraightJump,
    ForwardJump,
    BackJump,
    Down,
    FatalityReady
}

This will only help you for creating the Fatality move though.

For the rest of it, you'll need to add this to the Character Editor (move sets), in order to have a proper animation assigned (the standing dizzy animation).  And then you'll need to create a whole new case to trigger this new state.  Have a look in ControlsScript.cs, start with this function:

public void EndRound()

Good luck!  In theory, it shouldn't be too hard to implement this.  Depending on your skill level, it could take anywhere from a day to a week to work it all out.

Yeah, use State instead of SubState.  State is the primary condition the character is in, so it's ideal for a FatalityReady condition.

This line:

currentState = PossibleStates.Stand;

changes state to Stand.

This code:

if (currentState == PossibleStates.Down) {
    // I'm down, so something...
}

check's state is Down, at which point it does something.

I recommend the Unity Lessons (specifically, Scripting) to better understand how to code in Unity.