First, you need to know / understand that any number is a potential date / time for Excel and that Excel knows only datetime (and not date and time separately).
Here is why: anything before the decimals-placeholder are the days since December 31, 1899. So, the number 42,000.00 is also a date. It is the 42.000th day after December 31, 1899 which is December 27, 2014.
Anything after the decimal-placeholder is considered a fraction of a day. So, the value 0.5 is half a day and therefore the time 12 noon while 0.75 is 6 in the afternoon (pm).
Hence, all of the dates you are showing in your above examples may have also times associated with them and all the times you are showing might also have a date associated to them. Just change the number format on all of these cells to general numbers and you should see only Integer values for the dates (without any decimal places) while all the times should contain only be a fraction of 1 and should all have a 0 before the decimal-placeholder.
If that is not the case, then you have just found your culprit.
With the above knowledge it is also easy to calculate the difference between two dates or times. The difference is a simple addition or subtraction: 42,000.00 + 100 + 0.5 = 100 days after December 27, 2014 at 12 noon = April 6th, 2015 @ noon. So, you can just add the numbers and subtract them as you have done. If you wish to subtract two days to account for a weekend then you can simple subtract 2.
Now, let’s have a look at your above example:
June 23, 2016 converts to 42,544 while 07:41 in the morning converts to 0.32. So, we have here 42,544.32 while the ending datetime converts to 42,545.47.
The difference is calculated correctly as 1.15. That means that there is a difference of one day and the fraction of a day of 0.15 * 24 (hours in a day) = 3 hours and 36 minutes.
I guess the real problem you are having is to show the result correctly. If you show it as a date then it is one day after December 31, 1899 = January 1st, 1900. If you show the result as a time then you only get to see the 03:36:00. The best is probably to show the difference as a number only as 1.15. If you want to see the result differently then you will have to adjust the resulting view to your needs. For example:
= 1.15 * 24 = 27.6 hours difference between the start date/time and the end date/time.
Let me know if you have any further questions.