To customize one Button only set android:theme="@style/AppTheme.Button"
to your Button.
<Button
android:theme="@style/AppTheme.Button"
/>
Define your style as you did in your question
<style name="AppTheme.Button" parent="Base.Widget.AppCompat.Button">
<item name="colorButtonNormal">@color/my_color</item>
</style>
[EDIT]
See GitHub demo here