What are `scrolling boxes`?

As @alex said a scrolling box is a box where the value of overflow is set to a value different from visible (the default one). I cannot confirm but I concluded this based on this previous answer where overflow is creating some issues with sticky element.

As I explained there, if you have an element with overflow:hidden as an ancestor of the position:sticky this one will stop working because its offset will be calculated based on that box (with overflow:hidden) because it’s the nearest ancestor with a scrolling box. Since we used hidden we cannot scroll this box so we cannot see the sticky behavior.

Here is a basic example:

.wrapper {
  height:200vh;
  border:2px solid;
}
.wrapper >div {
  position:sticky;
  top:0;
  height:20px;
  background:red;
}

<div class="wrapper">
  <div></div>
</div>

In the below example, the viewport will be used for the reference because we have no scrolling box. Now let’s add overflow to the wrapper:

.wrapper {
  height:200vh;
  border:2px solid;
  overflow:scroll;
}
.wrapper >div {
  position:sticky;
  top:0;
  height:20px;
  background:red;
}

<div class="wrapper">
  <div></div>
</div>

Now our sticky element will consider the wrapper for the reference but since we don’t have any overflow, we won’t have any scroll so there is no way to trigger the sticky behavior. Also scrolling the viewport will do nothing.

If you add some overflow by adding another element inside we can trigger the sticky behavior:

.wrapper {
  height:200vh;
  border:2px solid;
  overflow:scroll;
  position:relative;
}
.wrapper >div {
  position:sticky;
  top:0;
  height:20px;
  background:red;
}

.wrapper > span {
  position:absolute;
  top:100%;
  height:50px;
  left:0;
  right:0;
  background:blue;
}

<div class="wrapper">
  <div></div>
  <span></span>
</div>

We can clearly see how the scroll of the wrapper is controlling the sticky element and the scroll of the viewport is doing nothing thus we can conclude that our wrapper is the the nearest ancestor with a scrolling box

Considering the last issue in the specification we can also read that:

Sticky positioning should really be defined in terms of the nearest scrollable ancestor, but there is currently no such term defined elsewhere in CSS. CSSOM View refers to “scrolling boxes.”

So probably a scrollable ancestor is the same as an ancestor with a scrolling box.