Unfortunately HTML5 does not provide an out-of-the-box way to do that.
However, using jQuery, you can easily control if a checkbox group has at least one checked element.
Consider the following DOM snippet:
<div class="checkbox-group required">
<input type="checkbox" name="checkbox_name[]">
<input type="checkbox" name="checkbox_name[]">
<input type="checkbox" name="checkbox_name[]">
<input type="checkbox" name="checkbox_name[]">
</div>
You can use this expression:
$('div.checkbox-group.required :checkbox:checked').length > 0
which returns true
if at least one element is checked.
Based on that, you can implement your validation check.