I have position but z index is not working

You need to remove the transform and replace it with something else and you will be able to move the pseudo-element behind:

:root{
  --size:200px;
}
#background {
  width:100%;
  height:100%;
  position:absolute;
  top:0;
  left:0;
  background: linear-gradient(-23.5deg, #000033, #00001a);
  z-index:-2;
}

#background #mainplanet {
  width:var(--size);
  height:var(--size);
  background:#fff;
  position:relative;
  top:calc(50% - var(--size)/2);
  left:calc(50% - var(--size)/2);
  border-radius:50%;
}

#background #mainplanet:before,#background #mainplanet:after{
  content:"";
  width:calc(var(--size) * 1.5);
  height:calc(var(--size) / 2);
  border:30px solid #000;
  position:absolute;
  top:10px;
  left:-80px;
  border-radius:50%;
  transform: rotateX(66deg) rotateY(170deg);
}

#background #mainplanet:before{
  border-top-color:transparent;
}

#background #mainplanet:after{
  z-index:-3;
}

<div id="background">
  <div id="mainplanet">
  </div>
</div>

As we may read in the specification:

8. All positioned, opacity or transform descendants, in tree order that fall into the following categories:
   1. All positioned descendants with ‘z-index: auto’ or ‘z-index: 0’, in tree order. For those with ‘z-index: auto’, treat the element as if it created a new stacking context, but any positioned descendants and descendants which actually create a new stacking context should be considered part of the parent stacking context, not this new one. For those with ‘z-index: 0’ treat the stacking context generated atomically.
   2. All opacity descendants with opacity less than 1, in tree order, create a stacking context generated atomically.
   3. All transform descendants with transform other than none, in tree order, create a stacking context generated atomically.

So the trick here is to avoid the positioned pseudo-element to belong to its parent stacking context to able to place it considering an upper stacking context thus it can be placed behind its parent.

So the parent element should not have z-index specified, opacity less than 1, use transform, etc.

Full list of the properties that create stacking context.