When you hover over an image in Publii the red trash can is displayed. I think this icon should only be displayed after you have clicked on the three dots at the right side? This way you also prevent the accidental removal of the caption and the alternative text.
Small feature request
It would be great if it was also possible to double click on an image to open the image/block editor (as an alternative of clicking on the three dots). Not really important though; only convenient.