Flexbox defines this as
For every unfrozen item on the line, multiply its flex shrink factor
by its inner flex base size, and note this as its scaled flex
shrink factor. Find the ratio of the item’s scaled flex shrink
factor to the sum of the scaled flex shrink factors of all
unfrozen items on the line. Set the item’s target main size to
its flex base size minus a fraction of the absolute value of the
remaining free space proportional to the ratio.
Simplifying, frozen flex items are those which can’t or don’t have to be flexed anymore. I will assume no min-width restrictions and non-zero flex shrink factors. This way all flex items are initially unfrozen and they all become frozen after only one iteration of the flex loop.
The inner flex base size depends on the value of box-sizing, defined by CSS2UI as
content-box: The specified width and height (and respective min/max properties) apply to the width and height respectively of the
content box of the element. The padding and border of the element are
laid out and drawn outside the specified width and height.
border-box: Length and percentages values for width and height (and respective min/max properties) on this element determine the
border box of the element. That is, any padding or border specified on
the element is laid out and drawn inside this specifiedwidth
andheight. The content width and height are calculated by
subtracting the border and padding widths of the respective sides from
the specified width and height properties. […] Used values, as
exposed for instance through getComputedStyle(), also refer to the
border box.
Basically, that means that sizes (widths, flex bases) have an inner an an outer variant. The inner size includes only the content, the outer one also includes paddings and border widths. The length specified in the stylesheet will be used as the inner size in case of box-sizing: content-box, or as the outer one in case of box-sizing: border-box. The other can be calculated by adding or subtracting border and padding widths.
Neglecting lots of details, the algorithm would be something like
let sumScaledShrinkFactors = 0,
remainingFreeSpace = flexContainer.innerMainSize;
for (let item of flexItems) {
remainingFreeSpace -= item.outerFlexBasis;
item.scaledShrinkFactor = item.innerFlexBasis * item.flexShrinkFactor;
sumScaledShrinkFactors += item.scaledShrinkFactor;
}
for (let item of flexItems) {
let ratio = item.scaledShrinkFactor / sumScaledShrinkFactors;
item.innerWidth = item.innerFlexBasis + ratio * remainingFreeSpace;
}
With no paddings, it’s as you explain
(width)
innerW │ padd │ outerW
───────┼──────┼───────
300px * (1 + 2 / 1000px * -200px) = 180px │ 0px │ 180px
200px * (1 + 1 / 1000px * -200px) = 160px │ 0px │ 160px
100px * (1 + 2 / 1000px * -200px) = 60px │ 0px │ 60px
───────┼──────┼───────
400px │ 0px │ 400px
With 10px horizontal paddings, the available space is reduced by 3 * 2 * 10px = 60px, so now it’s -260px.
(width)
innerW │ padd │ outerW
───────┼──────┼───────
300px * (1 + 2 / 1000px * -260px) = 144px │ 20px │ 164px
200px * (1 + 1 / 1000px * -260px) = 148px │ 20px │ 168px
100px * (1 + 2 / 1000px * -260px) = 48px │ 20px │ 68px
───────┼──────┼───────
340px │ 60px │ 400px
When you use box-sizing: border-box, the specified flex bases are the outer ones, so the paddings are subtracted from them to calculate the inner ones, which are 280px, 180px, 80px. Then the sum of the scaled flex shrink factors becomes 2*280px + 180px + 2*80px = 900px. The available space is like in the case with no padding, because the outer flex bases are the same. Note the width retrieved by getComputedStyle will now be the outer one, so paddings are added back at the end.
(width)
innerW │ padd │ outerW
────────┼──────┼────────
280px * (1 + 2 / 900px * -200px) ≈ 155.6px │ 20px │ 175.6px
180px * (1 + 1 / 900px * -200px) = 140.0px │ 20px │ 160.0px
80px * (1 + 2 / 900px * -200px) ≈ 44.4px │ 20px │ 64.4px
────────┼──────┼────────
340.0px │ 60px │ 400.0px