How to find the coordinates of the buttons on a canvas, and click on them after using Java and Selenium?

The <canvas> element is within an <iframe>. So to invoke click() on the elements within the <canvas> you have to:

• Induce WebDriverWait for the desired frame to be available and switch to it.
• Induce WebDriverWait for the desired element to be clickable.

• You can use the following solution:

  • Code Block:

    driver.get("https://www.online-calculator.com/full-screen-calculator/")
    new WebDriverWait(driver, 10).until(ExpectedConditions.frameToBeAvailableAndSwitchToIt(By.id("fullframe")));
    WebElement canvas = new WebDriverWait(driver, 10).until(ExpectedConditions.elementToBeClickable(By.id("canvas")));
    

---

HTML5 Canvas

The element is only a container for graphics and is a rectangular area on an HTML page. By default, a canvas has no border and no content. However, an id attribute (to be referred to in a script), a width and height attribute are specified to define the size of the canvas. To add a border, the style attribute is used. An example:

<canvas id="myCanvas" width="200" height="100" style="border:1px solid #000000;"></canvas>

The HTML canvas is a two-dimensional grid. The upper-left corner of the canvas has the coordinates (0,0).

In the article Automated Testing of HTML5 Canvas Applications with Selenium WebDriver @Aaron Mulder mentions, interacting with the elements within <canvas> is possible using event support of the Actions Class API:

• moveToElement(WebElement target, int xOffset, int yOffset): Moves the mouse to an offset from the top-left corner of the element.

  new Actions(driver).moveToElement(canvas, xWithinCanvas, yWithinCanvas).click().perform();
  

But is not 100% reliable as, every mouse down, mouse up, or mouse click happens at the center of the element. So the code above produces a mouse move event to the provided coordinates (x,y), then a mouse move event to the center of the <canvas>, then a mouse down, mouse up, and click at the center of the <canvas>. That should have been fine for a <button> but is not worth for a <canvas>, where you want to be able to click at a specific location.

The workaround, is to dispatch synthesized mouse events using JavaScript as follows:

// pageX and pageY are offsets which you need to know through mouse coordinates.
((JavascriptExecutor)driver).executeScript("var evt = $.Event('click', { pageX: " + x +
   ", pageY: " + (y + 55) + " } );" +
   "$('#myCanvas').trigger(evt);");

However, to click on the elements within the <canvas> you can be at ease using firefox as the mouse move event works well in Firefox and you can avoid using the mouse coordinates as the event processing as follows:

new Actions(driver).moveToElement(
   canvas, xWithinCanvas, yWithinCanvas).perform();
((JavascriptExecutor)driver).executeScript("$('#canvas').click();");

---

This usecase

To automate a substruction operation e.g. 3-1= using Selenium you can use the following solution:

• Code Block:

  driver.get("https://www.online-calculator.com/full-screen-calculator/");
  new WebDriverWait(driver, 10).until(ExpectedConditions.frameToBeAvailableAndSwitchToIt(By.id("fullframe")));
  WebElement canvas = new WebDriverWait(driver, 10).until(ExpectedConditions.elementToBeClickable(By.id("canvas")));
  //clicking on 3
  new Actions(driver).moveToElement(canvas, 0, 0).moveByOffset(0,(255/6)*3).click().build().perform();
  //clicking on the substract sign (-)
  new Actions(driver).moveToElement(canvas, 0, 0).moveByOffset((174/5)*2,(255/6)*3).click().build().perform();
  //clicking on 1
  new Actions(driver).moveToElement(canvas, 0, 0).moveByOffset(-(174/5)*4,(255/6)*3).click().build().perform();
  //clicking on equals to sign (=)
  new Actions(driver).moveToElement(canvas, 0, 0).moveByOffset((174/5)*4,(255/6)*4).click().build().perform();
  

• Execution Video:

---

Reference

You can find a couple of relevant detailed discussion in:

• How to perform Mouse Wheel scrolling over HTML5 Canvas in Selenium?