atoi vs atol vs strtol vs strtoul vs sscanf

which function would be best to convert either a decimal, hexadecimal, or octal number to an int the best (?)

To convert such text to int, recommend long strtol(const char *nptr, char **endptr, int base); with additional tests when converting to int, if needed.

Use 0 as the base to assess early characters in steering conversion as base 10, 16 or 8.
@Mike Holt

Convert text per:
Step 1: Optional whitespaces like `' '`, tab, `'\n'`, ... .
Step 2: Optional sign: `'-'` or `'+'`.
Step 3:
  0x or 0X followed by hex digits--> hexadecimal  
  0 --> octal  
  else --> decimal  

Sample code

#include <errno.h>
#include <limits.h>
#include <stdlib.h>

int mystrtoi(const char *str) {
  char *endptr;
  errno = 0;
  //                                   v--- determine conversion base
  long long_var = strtol(str, &endptr, 0);
  //   out of range   , extra junk at end,  no conversion at all   
  if (errno == ERANGE || *endptr != '\0' || str == endptr) {
    Handle_Error();
  }

  // Needed when `int` and `long` have different ranges
  #if LONG_MIN < INT_MIN || LONG_MAX > INT_MAX
  if (long_var < INT_MIN || long_var > INT_MAX) {
    errno = ERANGE;
    Handle_Error();
  }
  #endif

  return (int) long_var;
}

---

atoi vs atol vs strtol vs strtoul vs sscanf … to int

atoi()
Pro: Very simple.
Pro: Convert to an int.
Pro: In the C standard library.
Pro: Fast.
Con: On out of range errors, undefined behavior. @chqrlie
Con: Handle neither hexadecimal nor octal.

atol()
Pro: Simple.
Pro: In the C standard library.
Pro: Fast.
Con: Converts to an long, not int which may differ in size.
Con: On out of range errors, undefined behavior.
Con: Handle neither hexadecimal nor octal.

strtol()
Pro: Simple.
Pro: In the C standard library.
Pro: Good error handling.
Pro: Fast.
Pro: Can handle binary. (base 2 to base 36)
Con: Convert to an long, not int which may differ in size.

strtoul()
Pro: Simple.
Pro: In the C standard library.
Pro: Good error handling.
Pro: Fast.
Pro: Can handle binary.
—: Does not complain about negative numbers.
Con: Converts to an unsigned long, not int which may differ in size.

sscanf(..., "%i", ...)
Pro: In the C standard library.
Pro: Converts to int.
—: Middle-of-the-road complexity.
Con: Potentially slow.
Con: OK error handling (overflow is not defined).

All suffer/benefit from locale settings. §7.22.1.4 6 “In other than the “C” locale, additional locale-specific subject sequence forms may be accepted.”

---

Additional credits:
@Jonathan Leffler: errno test against ERANGE, atoi() decimal-only, discussion about errno multi-thread concern.
@Marian Speed issue.
@Kevin Library inclusiveness.

---

For converting short, signed char, etc., consider strto_subrange().