You can use a discrete colormap and modify the colorbar, instead of using a legend.
value_to_int = {j:i for i,j in enumerate(pd.unique(df.values.ravel()))} # like you did
n = len(value_to_int)
# discrete colormap (n samples from a given cmap)
cmap = sns.color_palette("Pastel2", n)
ax = sns.heatmap(df.replace(value_to_int), cmap=cmap)
# modify colorbar:
colorbar = ax.collections[0].colorbar
r = colorbar.vmax - colorbar.vmin
colorbar.set_ticks([colorbar.vmin + r / n * (0.5 + i) for i in range(n)])
colorbar.set_ticklabels(list(value_to_int.keys()))
plt.show()
The colorbar part is adapted from this answer
HTH