4:3 to 16:9 non-linear stretch tool(s) similar to superview

[sickgreg]

Hi,

I have just joined up and would like to share my non-linear stretch tool allowing you to watch in real time your full image sensor height videos without cropping the top and bottom. Many cameras do not have this feature built in.

Realtime-Superview-Stretch

https://github.com/sickgreg/Realtime...retch/releases

An HLSL (High Level Shader Language) pixel shader that works in HLSL supported players like Media Player Classic (MPC-HC) to apply a non-linear horizontal stretch (actually exponential, reversing the lens' fisheye squishy effect at the edges) in real time. This allows you to get the absolute most out of your 4:3 sensor and works great in my case for FPV drone footage.

I'm not a programmer but i "needed" this and went at it boots and all. I was inspired by the fantastic https://github.com/Niek/superview video converter tool which of course re-encodes the whole video on your CPU, great if you want to upload to YouTube, but not so great for just reviewing (in my case) the flight footage.

Please enjoy

Feel free to edit the code. It's a miracle that I even got it working using basic linear interpolation as the method of remapping pixels. The stretch can look a bit blocky because of this method. It is beyond my ability to utilize a nicer resize algorithm but probably not rocket science either. That being said, it still looks really great, especially if added to your player post resize on a large monitor.

Cheers
Greg

[nutsey]

Hi,

Thanks for sharing!

What about adding another algorithm where 16:6 picture is not stretched from the sides but is nonlinearly squeezed along its horizontal?

I've implemented it as 'Ultraview' for Firefly 8SE custom fws here: https://www.goprawn.com/forum/ambare...1984#post11984

[sickgreg]

This is what it does.... Is that the same as yours?
Nonlinearly squeezed along its horizontal and scaled to 16:9 in this case by the player. Actually there is misleading code in my .hlsl file! Left over unused variable from an earlier version:

#define PERCENTAGE_NOT_STRETCHED 0.0

It's set to zero and hence not used

[nutsey]

Mine is not upscaled but downscaled nonlinearly in perpendicular dimension.

Like this:

[sickgreg]

[sickgreg]

I've been trying to create a Realtime-Nutsey-Squash.hlsl shader but it is doing it perfectly backwards.... squishing the center instead of the top and bottom and I can't figure out the math.... Any idea??

sampler s0 : register(s0);

float4 main(float2 tex : TEXCOORD0) : COLOR
{


    tex.y = tex.y * 2.0 - 1.0;
    // convert incoming pixel coord from a value 0 to 1 to a value -1 to +1

    float distanceFromEdge = (max(  abs(tex.y), 0.0 ) * 0.5 + 0.5 );
    // pixel gets its distanceFromEdge value between 0 and 1

    tex.y = lerp( tex.y / 0.75, tex.y,    pow(distanceFromEdge,2) );
    // linear interpolate using the square of the distance from the edge

    tex.y = tex.y * 0.5 + 0.5;
    // convert outgoing pixel coord from a value -1 to +1 back to a value 0 to 1

    float4 l = tex2D(s0, tex);
    return l;    //return the pixel 
}

[nutsey]

Actually my distortion profile is for 4:3 picture linearly resized into 16:9 (common modes for Ambarella cams with 4:3 CMOS sensors) and it stretches the center to get proper geometry

[sickgreg]

Okay so the Ambarella applies lens distortion correction that you have defined somehow...? Or is the profile already pre-defined? How do you define a distortion profile and do you know if the icatch v35ma has such a feature?

Obviously the methods I posted are applied either via re-encoding (superview.exe) or in real time whilst watching on the computer in Media Player Classic with realtime-superview-stretch.hlsl

Sorry about all the questions!

Cheers
Greg

[nutsey]

Talking about Ambarella there is a distortion correction table with shift values. By default it is calculated to compensate average barrel distortion (fullframe fisheye). I completely redisigned it to improve frame geometry after 4:3->16:9 conversion.

As for iCatch, I'm going to get my V35 cam in a couple of weeks. Hope it uses an SPI flash memory that can be dumped without any soldering.

[nutsey]

Hi sickgreg

I tried to use your shader with VirtualDub2 but it does not work. Here are some shaders that work: https://github.com/Loadus/virtualdub-shaders

To load a shader go to Options - Preferences - Display - Use Direct3D 9 - Use effect file ...

Is it possible to modify your code in order to use it with this software?

[nutsey]

I tried adding

technique {
pass {
PixelShader = compile ps_3_0 main();
}
}

but the video panes turned black.

[sickgreg]

Did you get it to work in a player such as MPC-HC or MPC-BE ?

It would be good in VirtualDub2 for processing files rather than only watching it live. I will have a look into that. I didn't even know virtual dub existed for anything other than AVI. That is cool VD2.

I got realtime-nutsey-squash.hlsl working by the way

// Realtime-nutsey-squash.hlsl by sickGreg
sampler s0 : register(s0);

float4 main(float2 tex : TEXCOORD0) : COLOR
{


tex.y = tex.y * 2.0 - 1.0;
// convert incoming pixel coord from a value 0 to 1 to a value -1 to +1

float distanceFromEdge = (max( abs(tex.y), 0.0 ) * 0.5 + 0.5 );
// pixel gets its distanceFromEdge value between 0 and 1

tex.y = lerp( tex.y * 0.75, tex.y, distanceFromEdge*distanceFromEdge);
// linear interpolate using the square of the distance from the edge

tex.y = tex.y * 0.5 + 0.5;
// convert outgoing pixel coord from a value -1 to +1 back to a value 0 to 1

float4 l = tex2D(s0, tex);
return l; //return the pixel
}

[sickgreg]

Like this!

realtime-superview-stretch




Or using realtime-nutsey-squash (Ultraview)



Give me a moment and i will write up how to do it.

Thanks Nutsey for pointing me in the direction of VirtualDub 2.

[sickgreg]

Put DirectX settings back to default

Download http://www.virtualdub.org/downloads/vdshader-1.4.zip and extract to VirtualDub2/plugins32 folder
Open 32-bit version of VirtualDub2 and open your video file
Click Video/Filter and add vdshader
Copy and paste the code into the vdshader window

//realtime-superview-stretch.hlsl by sickGreg
sampler s0 : register(s0);

float4 main(float2 tex : TEXCOORD0) : COLOR
{

// convert incoming pixel coord from a value 0 to 1 to a value -1 to +1
tex.x = tex.x * 2.0 - 1.0;

// give pixel its distanceFromEdge value between 0 and 1
float distanceFromEdge = max( abs(tex.x ), 0.0 ) * 0.5 + 0.5 ;

// linear interpolate using the square of the distance from the edge
tex.x = lerp( tex.x / 0.75, tex.x, pow(distanceFromEdge,2) );

// convert outgoing pixel coord from a value -1 to +1 back to a value 0 to 1
tex.x = tex.x * 0.5 + 0.5;

//return the pixel
float4 l = tex2D(s0, tex);
return l;
}

Save As "superview-stretch.fx"
Close the vdshader window and the shader should now be applied to the video preview
You'll also need to add the resize filter and resize the horizontal to 133.333333333%
I figure the results should be better resizing first and then non-linear stretching

[nutsey]

It works! Thank you!