Did some metrics around these in a single threaded model and here are the results I got.
~~~~~~~~~~~~~~~~~~Time Metrics~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~ Tokenizer | String.Split() | while+SubString | Scanner | ScannerWithCompiledPattern ~ ~ 4.0 ms | 5.1 ms | 1.2 ms | 0.5 ms | 0.1 ms ~ ~ 4.4 ms | 4.8 ms | 1.1 ms | 0.1 ms | 0.1 ms ~ ~ 3.5 ms | 4.7 ms | 1.2 ms | 0.1 ms | 0.1 ms ~ ~ 3.5 ms | 4.7 ms | 1.1 ms | 0.1 ms | 0.1 ms ~ ~ 3.5 ms | 4.7 ms | 1.1 ms | 0.1 ms | 0.1 ms ~ ____________________________________________________________________________________________________________
The out come is that Scanner gives the best performance, Now the same needs to be evaluated on a multithreaded mode ! One of my senior’s say that the Tokenizer gives a CPU spike and String.split does not.