how to implement natural pinch scaling of elements inside SVG

[bstro]

Hi there! I'm trying to get use-gesture/react working with SVG; that is, I would like to drag/pinch <image> elements around inside an <svg> container, and I'm running into trouble getting the pinch-to-scale behavior working correctly. It seems to have a mind of its own. I have a feeling this has to do with the fact the fact that I'm using SVG's viewBox parameter and need to apply the svg's scaleCtm value to some values inside the pinch handler (as I had to do for drag), but I'm not sure. Any ideas?

Sometimes it kinda works okay, but if you try pinching a few times you should quickly run into some wonkiness:
https://stackblitz.com/edit/vitejs-vite-amwj9n?file=src%2FApp.tsx%3AL12,src%2FImage.tsx%3AL43-L43&terminal=dev

I started from this use-gesture example since it's pretty much the behavior I'm looking for, but I need to adapt it to my SVG usecase.

I came across this but not exactly sure how to apply it to my situation: https://excalidraw.com/#json=cuumlwmDDlK3PkGuLvFdt,laMdKlcWUwuw5jSy4JaYFQ