C
C#•3mo ago
Romen

SkiaSharp - Optimal use of runtime effects

Hi! I have a video player that renders the video frames with a custom SKRuntimeEffect that is essentially a fragment shader. I have a SKRuntimeEffect that is created once when the video player is initialized using some SKSL that looks roughly like this: 
uniform shader image;
uniform float fTexWidth;
uniform float fTexHeight;

vec4 main(vec2 texCoord)
{
    // some custom image sampling & color work here
}
uniform shader image;
uniform float fTexWidth;
uniform float fTexHeight;

vec4 main(vec2 texCoord)
{
    // some custom image sampling & color work here
}
The problem I am having is that this shader seems to need a rebuild for every new image frame, because of how the SKSL pipeline works -- "textures" are abstracted as another shader that needs to be chained before my custom shader. So the code to do this looks something like this right now: 
// Update the uniforms for the current image size
_myEffectUniforms.Add("fTexWidth", (float)(image?.Width ?? 0));
_myEffectUniforms.Add("fTexHeight", (float)(image?.Height ?? 0));

// Attach the image as a "child" to the runtimeeffect
SKRuntimeEffectChildren children = new SKRuntimeEffectChildren(_myEffect);
if (image != null)
{
  children.Add("image", image.ToShader()); // This call is very expensive
}

// Rebuild the shader using the new child
_myPaint.Shader = _myEffect.ToShader(_myEffectUniforms, children);
// Update the uniforms for the current image size
_myEffectUniforms.Add("fTexWidth", (float)(image?.Width ?? 0));
_myEffectUniforms.Add("fTexHeight", (float)(image?.Height ?? 0));

// Attach the image as a "child" to the runtimeeffect
SKRuntimeEffectChildren children = new SKRuntimeEffectChildren(_myEffect);
if (image != null)
{
  children.Add("image", image.ToShader()); // This call is very expensive
}

// Rebuild the shader using the new child
_myPaint.Shader = _myEffect.ToShader(_myEffectUniforms, children);
The image.ToShader() call where I add the image as a "child" shader in the pipeline and this call takes a significant amount of time that limits the FPS of the video player. My questions: - Where the heck is the documentation for SKSL? I believe the solution I came up with is sub-optimal just because I can't find documentation and had to trial-and-error to get this to work at all. - In GLSL you would normally use a special kind of shader variable for textures which allows rebinding the texture without needing to recompile the shader just to change the image it uses. SKSL does not seem to have this capability because they have modelled textures as "shaders" in their pipeline and so textures have to be comverted to shaders before being inserted in the pipeline. Why does it work this way? Did I miss an API in skia(sharp) that allows me to re-bind the image without rebuilding the shader?
1 Reply
Romen
RomenOP•3mo ago
If anyone else stumbles across this, I have found that using SKImage instead of SKBitmap is much faster. This seems to take advantage of OpenGL texture buffers as the backing memory for the object instead of CPU memory. Then the .ToShader() call becomes something entirely different due to that.

Did you find this page helpful?