Done it finally by myself.
Draw everything by applying this formula to (px,py) coordinates:
float px = ev.getX() / mScaleFactor + rect.left;
float py = ev.getY() / mScaleFactor + rect.top;
rect = canvas.getClipBounds();
//Get them in on Draw function and apply above formula before drawing