Adder-Subtractor Circuit With Negative Results

So, I understand how binary arithmetic works, and I understand how an adder-subtractor works for signed numbers. There is only one thing I am not sure about:

All the cases work ok in the circuit I have, except if the result of a subtraction is negative, I need to take the two’s complement of the output byte to get the actual result. What can I do about it? Do I need an extra array of adders to compute the two’s complement only in that specific way, or is there any smarter solution I can apply?

Thanks in advance.