Exactly what it says:
'name1'
has a different data type to 'name' + CAST((rn+1) as varchar(255))
Try this (untested)
;with cte as
(
select 1 as rn, CAST('name1' as varchar(259)) as nm
union all
select rn+1,nm = 'name' + CAST((rn+1) as varchar(255))
from cte a where rn<10)
select * from cte
Basically, you have to ensure the length matches too. For the recursive bit, you may have to use CAST('name' AS varchar(4))
if it fails again