Two alternative ways of doing this:
1) with the dplyr package:
library(dplyr)
df1 <- df1 %>%
group_by(X) %>%
mutate(Y.new = mean(Y))
2) with the data.table package:
library(data.table)
setDT(df1)[, Y.new := mean(Y), by = X]
both give the following result:
> df1 X Y Y.new 1: A 1 2 2: A 2 2 3: A 3 2 4: B 4 5 5: B 5 5 6: B 6 5