How to avoid triggering suspense fallback when using ssr prefetch+suspense after first render.

I have a component that uses useSuspenseQuery and the page uses prefetchQuery. The project is a next.js app router app. The client component has buttons that changes the params used in the query. Every change triggers the suspense fallback. Is there anyway to just render the suspense fallback for the initial render?