Why #’ is used before lambda in Common Lisp?

#'foo is an abbreviation for (function foo) by the reader.

In CL, there are several different namespaces, #'foo or (function foo) will return the functional value of foo.

You may want to search for “Lisp-1 vs. Lisp-2”, check other Stackoverflow questions, or read an old article by Pitman and Gabriel in order to learn more about the concept of multiple namespaces (also called slots or cells of symbols).

The reason that, in the case of lambda, the #' may be omitted in CL is that it is a macro, which expands thusly (taken from the Hyperspec):

(lambda lambda-list [[declaration* | documentation]] form*)
==  (function (lambda lambda-list [[declaration* | documentation]] form*))
==  #'(lambda lambda-list [[declaration* | documentation]] form*)

#' may still be used for historic reasons (I think that in Maclisp lambdas didn’t expand to the function form), or because some people think, that tagging lambdas with sharpquotes may make the code more readable or coherent. There may be some special cases in which this makes a difference, but in general, it doesn’t really matter which form you choose.

I guess you can think of it like this: (function (lambda ...)) returns the function (lambda ...) creates. Note that lambda in the CL Hyperspec has both a macro AND a symbol entry. From the latter:

A lambda expression is a list that can
be used in place of a function name in
certain contexts to denote a function
by directly describing its behavior
rather than indirectly by referring to
the name of an established function.

From the documentation of function:

If name is a lambda expression, then a
lexical closure is returned.

I think the difference is also related to calling lambda forms like this: ((lambda ...) ...) where it is treated as a form to be evaluated, vs. (funcall #'(lambda ...) ...). If you want to read more on the topic, there is a c.l.l thread about it.

Some quotes from that thread:

(lambda (x) ... by itself is just some
unquoted list structure. It is its
appearance as an argument to the
FUNCTION special form (function
(lambda (x) ... that causes the
function object to exist

and:

It’s also compounded by the fact that
the LAMBDA macro was a rather late
addition the ANSI Common Lisp, so all
of the really old guys (i.e., like me)
learned their lisp when you needed to
supply the #’ to the lambda expression
in the mapping functions. Otherwise
the non-existent lambda function would
be invoked.

The macro addition changed that, but
some of us are too set in our ways to
want to change.