Well, I found it. It IS a bug in the framework and to work around it you add a call to UpdateLayout. Using statement can be changed to the following to provide a fix;
Using XPSItem As New Windows.Xps.Packaging.XpsDocument(PathToTestXps, System.IO.FileAccess.Read)
Dim FixedDocSequence As Windows.Documents.FixedDocumentSequence
Dim DocPager As Windows.Documents.DocumentPaginator
FixedDocSequence = XPSItem.GetFixedDocumentSequence
DocPager = FixedDocSequence.DocumentPaginator
DocPager.ComputePageCount()
' This is the fix, each page must be laid out otherwise resources are never released.'
For PageIndex As Integer = 0 To DocPager.PageCount - 1
DirectCast(DocPager.GetPage(PageIndex).Visual, Windows.Documents.FixedPage).UpdateLayout()
Next
FixedDocSequence = Nothing
End Using