What are the differences between strtok and strsep in C

by

One major difference between strtok() and strsep() is that strtok() is standardized (by the C standard, and hence also by POSIX) but strsep() is not standardized (by C or POSIX; it is available in the GNU C Library, and originated on BSD). Thus, portable code is more likely to use strtok() than strsep().

Another difference is that calls to the strsep() function on different strings can be interleaved, whereas you cannot do that with strtok() (though you can with strtok_r()). So, using strsep() in a library doesn’t break other code accidentally, whereas using strtok() in a library function must be documented because other code using strtok() at the same time cannot call the library function.

The manual page for strsep() at kernel.org says:

The strsep() function was introduced as a replacement for strtok(3), since the latter cannot handle empty fields.

Thus, the other major difference is the one highlighted by George Gaál in his answer; strtok() permits multiple delimiters between a single token, whereas strsep() expects a single delimiter between tokens, and interprets adjacent delimiters as an empty token.

Both strsep() and strtok() modify their input strings and neither lets you identify which delimiter character marked the end of the token (because both write a NUL '\0' over the separator after the end of the token).

When to use them?

  • You would use strsep() when you want empty tokens rather than allowing multiple delimiters between tokens, and when you don’t mind about portability.
  • You would use strtok_r() when you want to allow multiple delimiters between tokens and you don’t want empty tokens (and POSIX is sufficiently portable for you).
  • You would only use strtok() when someone threatens your life if you don’t do so. And you’d only use it for long enough to get you out of the life-threatening situation; you would then abandon all use of it once more. It is poisonous; do not use it. It would be better to write your own strtok_r() or strsep() than to use strtok().

Why is strtok() poisonous?

The strtok() function is poisonous if used in a library function. If your library function uses strtok(), it must be documented clearly.

That’s because:

  1. If any calling function is using strtok() and calls your function that also uses strtok(), you break the calling function.
  2. If your function calls any function that calls strtok(), that will break your function’s use of strtok().
  3. If your program is multithreaded, at most one thread can be using strtok() at any given time — across a sequence of strtok() calls.

The root of this problem is the saved state between calls that allows strtok() to continue where it left off. There is no sensible way to fix the problem other than “do not use strtok()“.

  • You can use strsep() if it is available.
  • You can use POSIX’s strtok_r() if it is available.
  • You can use Microsoft’s strtok_s() if it is available.
  • Nominally, you could use the ISO/IEC 9899:2011 Annex K.3.7.3.1 function strtok_s(), but its interface is different from both strtok_r() and Microsoft’s strtok_s().

BSD strsep():

char *strsep(char **stringp, const char *delim);

POSIX strtok_r():

char *strtok_r(char *restrict s, const char *restrict sep, char **restrict state);

Microsoft strtok_s():

char *strtok_s(char *strToken, const char *strDelimit, char **context);

Annex K strtok_s():

char *strtok_s(char * restrict s1, rsize_t * restrict s1max,
               const char * restrict s2, char ** restrict ptr);

Note that this has 4 arguments, not 3 as in the other two variants on strtok().

More Related Contents:

  1. How does strtok() split the string into tokens in C?
  2. Using strtok() in nested loops in C?
  3. Why is strtok() Considered Unsafe?
  4. Nested strtok function problem in C [duplicate]
  5. strtok function thread safety
  6. Using strtok in c
  7. Split string into tokens and save them in an array
  8. C: correct usage of strtok_r
  9. How to split a string to 2 strings in C
  10. Need to know when no data appears between two token separators using strtok()
  11. strtok segmentation fault
  12. C’s strtok() and read only string literals
  13. Why is strtok changing its input like this?
  14. Error: void value not ignored as it ought be
  15. Why does sizeof(x++) not increment x?
  16. How can I multiply and divide using only bit shifting and adding?
  17. Correct way of passing 2 dimensional array into a function
  18. In a C expression where unsigned int and signed int are present, which type will be promoted to what type?
  19. Why is #include not required to use printf()?
  20. How to convert ASCII character to CGKeyCode?
  21. How to merge two windows vc static library into one
  22. Overriding C library functions, calling original
  23. Sizeof vs Strlen
  24. Linux/POSIX equivalent for Win32’s CreateEvent, SetEvent, WaitForSingleObject
  25. The difference between asm, asm volatile and clobbering memory
  26. To check the E2BIG error condition in exec
  27. C Programming – XOR Bitwise Operation
  28. How to convert integer to char in C? [duplicate]
  29. Scan multiple integers without knowing the actual number of integers
  30. Can I compare int with size_t directly in C?

Leave a Comment