Tuesday, 1 January 2019

Does H.264 encoded video with BT.709 matrix include any gamma adjustment?

I have read the BT.709 spec a number of times and the thing that is just not clear is should an encoded H.264 bitstream actually apply any gamma curve to the encoded data? Note the specific mention of a gamma like formula in the BT.709 spec. Apple provided examples of OpenGL or Metal shaders that read YUV data from CoreVideo provided buffers do not do any sort of gamma adjustment. YUV values are being read and processed as though they are simple linear values. I also examined the source code of ffmpeg and found no gamma adjustments being applied after the BT.709 scaling step. I then created a test video with just two linear grayscale colors 5 and 26 corresponding to 2% and 10% levels. When converted to H.264 with both ffmpeg and iMovie, the output BT.709 values are (YCbCr) (20 128 128) and (38 128 128) and these values exactly match the output of the BT.709 conversion matrix without any gamma adjustment.

A great piece of background on this topic can be found at Quicktime Gamma Bug. It seems that some historical issues with Quicktime and Adobe encoders were improperly doing different gamma adjustments and the results made video streams look awful on different players. This is really confusing because if you compare to sRGB, it clearly indicates how to apply a gamma encoding and then decode it to convert between sRGB and linear. Why does BT.709 go into so much detail about the same sort of gamma adjustment curve if no gamma adjustment is applied after the matrix step when creating a h.264 data stream? Are all the color steps in a h.264 stream meant to be coded as straight linear (gamma 1.0) values?

In case specific example input would make things more clear, I am attaching 3 color bar images, the exact values of different colors can be displayed in an image editor with these image files.

This first image is in the sRGB colorspace and is tagged as sRGB.

sRGB colorspace

This second image has been converted to the linear RGB colorspace and is tagged with a linear RGB profile.

linear RGB colorspace

This third image has been converted to REC.709 profile levels with Rec709-elle-V4-rec709.icc from elles_icc_profiles . This seems to be what one would need to do to simulate "camera" gamma as described in BT.709.

BT.709 colorspace ICC

Note how the sRGB value in the lower right corner (0x555555) becomes linear RGB (0x171717) and the BT.709 gamma encoded value becomes (0x464646). What is unclear is if I should be passing a linear RGB value into ffmpeg or if I should be passing an already BT.709 gamma encoded value which would then need to be decoded in the client before the linear conversion Matrix step to get back to RGB.

Update:

Based on the feedback, I have updated my C based implementation and Metal shader and uploaded to github as an iOS example project MetalBT709Decoder.

Encoding a normalized linear RGB value is implemented like this:

static inline
int BT709_convertLinearRGBToYCbCr(
                            float Rn,
                            float Gn,
                            float Bn,
                            int *YPtr,
                            int *CbPtr,
                            int *CrPtr,
                            int applyGammaMap)
{
  // Gamma adjustment to non-linear value

  if (applyGammaMap) {
    Rn = BT709_linearNormToNonLinear(Rn);
    Gn = BT709_linearNormToNonLinear(Gn);
    Bn = BT709_linearNormToNonLinear(Bn);
  }

  // https://www.itu.int/dms_pubrec/itu-r/rec/bt/R-REC-BT.709-6-201506-I!!PDF-E.pdf

  float Ey = (Kr * Rn) + (Kg * Gn) + (Kb * Bn);
  float Eb = (Bn - Ey) / Eb_minus_Ey_Range;
  float Er = (Rn - Ey) / Er_minus_Ey_Range;

  // Quant Y to range [16, 235] (inclusive 219 values)
  // Quant Eb, Er to range [16, 240] (inclusive 224 values, centered at 128)

  float AdjEy = (Ey * (YMax-YMin)) + 16;
  float AdjEb = (Eb * (UVMax-UVMin)) + 128;
  float AdjEr = (Er * (UVMax-UVMin)) + 128;

  *YPtr = (int) round(AdjEy);
  *CbPtr = (int) round(AdjEb);
  *CrPtr = (int) round(AdjEr);

  return 0;
}

Decoding from YCbCr to linear RGB is implemented like so:

static inline
int BT709_convertYCbCrToLinearRGB(
                             int Y,
                             int Cb,
                             int Cr,
                             float *RPtr,
                             float *GPtr,
                             float *BPtr,
                             int applyGammaMap)
{
  // https://en.wikipedia.org/wiki/YCbCr#ITU-R_BT.709_conversion
  // http://www.niwa.nu/2013/05/understanding-yuv-values/

  // Normalize Y to range [0 255]
  //
  // Note that the matrix multiply will adjust
  // this byte normalized range to account for
  // the limited range [16 235]

  float Yn = (Y - 16) * (1.0f / 255.0f);

  // Normalize Cb and CR with zero at 128 and range [0 255]
  // Note that matrix will adjust to limited range [16 240]

  float Cbn = (Cb - 128) * (1.0f / 255.0f);
  float Crn = (Cr - 128) * (1.0f / 255.0f);

  const float YScale = 255.0f / (YMax-YMin);
  const float UVScale = 255.0f / (UVMax-UVMin);

  const
  float BT709Mat[] = {
    YScale,   0.000f,  (UVScale * Er_minus_Ey_Range),
    YScale, (-1.0f * UVScale * Eb_minus_Ey_Range * Kb_over_Kg),  (-1.0f * UVScale * Er_minus_Ey_Range * Kr_over_Kg),
    YScale, (UVScale * Eb_minus_Ey_Range),  0.000f,
  };

  // Matrix multiply operation
  //
  // rgb = BT709Mat * YCbCr

  // Convert input Y, Cb, Cr to normalized float values

  float Rn = (Yn * BT709Mat[0]) + (Cbn * BT709Mat[1]) + (Crn * BT709Mat[2]);
  float Gn = (Yn * BT709Mat[3]) + (Cbn * BT709Mat[4]) + (Crn * BT709Mat[5]);
  float Bn = (Yn * BT709Mat[6]) + (Cbn * BT709Mat[7]) + (Crn * BT709Mat[8]);

  // Saturate normalzied linear (R G B) to range [0.0, 1.0]

  Rn = saturatef(Rn);
  Gn = saturatef(Gn);
  Bn = saturatef(Bn);

  // Gamma adjustment for RGB components after matrix transform

  if (applyGammaMap) {
    Rn = BT709_nonLinearNormToLinear(Rn);
    Gn = BT709_nonLinearNormToLinear(Gn);
    Bn = BT709_nonLinearNormToLinear(Bn);
  }

  *RPtr = Rn;
  *GPtr = Gn;
  *BPtr = Bn;

  return 0;
}

I believe this logic is implemented correctly, but I am having a very difficult time validating the results. When I generate a .m4v file that contains gamma adjusted color values (osxcolor_test_image_24bit_BT709.m4v), the result come out as expected. But a test case like (bars_709_Frame01.m4v) that I found here does not seem to work as the color bar values seem to be encoded as linear (no gamma adjustment).

For a SMPTE test pattern, the 0.75 graylevel is linear RGB (191 191 191), should this RGB be encoded with no gamma adjustment as (Y Cb Cr) (180 128 128) or should the value in the bitstream appear as the gamma adjusted (Y Cb Cr) (206 128 128)?



from Does H.264 encoded video with BT.709 matrix include any gamma adjustment?

Monday, 31 December 2018

Where can I set contentOffset to the last item in UICollectionView before the view appears?

I am scrolling multiple horizontal collectionViews to the last (Far right) item using:

 timelineCollectionView.scrollToMaxContentOffset(animated: false)
   postsCollectionView.scrollToMaxContentOffset(animated: false)

this works great, except I can't figure out where to put them.

Various places I've tried:

viewWillAppear - it doesn't scroll as though the cells aren't fully loaded yet.

viewDidAppear - it does scroll perfectly. Awesome! Except now you can see it scroll even when you put animated: false. The collectionView loads at the far left for a split second before updating to the far right

viewDidLayoutSubviews - this works perfectly - timing and everything! However, it gets called many times. If I could determine which subview was just laid out, perhaps I could scroll it in there but Im not sure how.

Is there a better option? How can I do this?

Also these are the functions I am using to set content offset:

extension UIScrollView {

    var minContentOffset: CGPoint {
        return CGPoint(
            x: -contentInset.left,
            y: -contentInset.top)
    }

    var maxContentOffset: CGPoint {
        return CGPoint(
            x: contentSize.width - bounds.width + contentInset.right,
            y: contentSize.height - bounds.height + contentInset.bottom)
    }

    func scrollToMinContentOffset(animated: Bool) {
        setContentOffset(minContentOffset, animated: animated)
    }

    func scrollToMaxContentOffset(animated: Bool) {
        setContentOffset(maxContentOffset, animated: animated)
    }
}



from Where can I set contentOffset to the last item in UICollectionView before the view appears?

Simple Node/Express app not recognizing Session store

I have an extremely small express app to illustrate a problem I'm having.

I'm using connect-redis as a session store on my express app. I'm having a problem simply connecting to it though. Simply printing out req.session.store results in undefined as shown below:

const session = require('express-session')
const app = require('express')()
const RedisStore = require('connect-redis')(session);

const isDev = process.env.NODE_ENV !== 'production'

app.use(session({
  store: new RedisStore({
    host: 'localhost',
    port: 6379
  }),
  secret: 'super-secret-key', // TBD: grab from env
  resave: false,
  saveUninitialized: false,
  cookie: {
    maxAge: 1000 * 60 * 60 * 24,
    secure: !isDev, // require HTTPS in production
  }
}))

app.get('/', (req, res) => {
  console.log('**testing**')
  console.log(req.session.store)
  res.send('rendering this text')
})

app.listen(3001, () => {
  console.log('listening on 3001')
})


The output of this is:

listening on 3001
**testing**
undefined

In the store property, I've tried connecting via the url property and the client property in addition to the current host and port method.

I'm not seeing any errors either so I'm not sure why the store is always undefined.

What am I missing with this?

also, I did start redis using redis-server and I am able to connect to it through other clients


Update 1
Also, I did look at this StackOverflow which some found useful. The issue is, from version 1.5> of express-session, it says you don't need the cookie-parser module, so I don't believe that's the issue.



from Simple Node/Express app not recognizing Session store