Unhelpful Answer
There is not, and will never be, a way to compare two functions for equality. There is a mathematical proof that it is not possible in general.
“Pragmatic” approaches will either depend on the internal workings of your compiler (e.g. if two functions are equal if they are represented by the same memory address internally), or be less helpful than expected.
What is the expected result of succ == (+1)? How about let a == 1 in succ == (+a)?
I can’t think of a way to define (==) for functions that always makes sense, and I believe neither can anyone else.
Things Irrelevant to the Question
The type signature is missing a return type.
Your function has a rank 2 type, which is not standard Haskell 98, and, more importantly, is not necessary in this case. You don’t care if the passed-in function can deal with any Enum type, you only care that it works for Rank:
search :: (Rank -> Rank) -> Card -> [Card] -> [Card] -- much simpler.
I’d try to use case more often, and if/then less often. In particular, I’d write:
case [ n | n <- list, rank n == op (rank x)] of
[y] -> x : search op y list -- length == 1
_ -> [] -- otherwise
My Real Answer
Your function basically only works with two different values for op, but its type doesn’t say so; that doesn’t feel right for me.
You could do it the old-fashioned way:
data Direction = Succ | Pred deriving(Eq)
search :: Direction -> Card -> [Card] -> [Card]
search dir x list = if (dir == Succ && rank x == King) ...
... let op = case Direction of Succ -> succ ; Pred -> pred
in ...
Or you could make the parameter more general, and pass a function that may fail gracefully (by returning Nothing) instead:
maybeSucc x | x == maxBound = Nothing
| otherwise = Just (succ x)
search :: (Rank -> Maybe Rank) -> Card -> [Card] -> [Card]
search op x list = case op (rank x) of
Nothing -> []
Just theRank -> case [ n | n <- list, rank n == theRank ] of ...