The problem is not with your CSS but with your SVG. The SVG will grow to fill the entire element box’s background (as expected). How your SVG scale then becomes the controlling factor:
Set a viewBox="0 0 width height"
(in pixels) attribute on your <svg>
element and remove its width
and height
attributes. You also need to set preserveAspectRatio="xMinYMid"
(x/vertically left-aligned, y/horizontally middle-aligned) on the svg
element. This works with Internet Explorer 10 and 11, at least.
<svg viewbox="0 0 64 64"
preserveAspectRatio="xMinYMid">
… </svg>
Learn more about the preserveAspectRatio and viewBox attributes. Source, “Getting started with SVG” in the IEblog.