Bugs and Suggestions

Fix, part 4/4: MPEG-4 Part 2 esds (MediaPlayer/Sources/FFMpegMediaFrameSourceContextHelpers.swift)

static func mpeg4EsdsAtom(_ extradata: Data) -> Data {
if extradata.count > 5 && extradata.prefix(4) == Data([0, 0, 0, 0]) && extradata[extradata.startIndex + 4] == 0x03 {
return extradata
}
func descriptor(_ tag: UInt8, _ payload: Data) -> Data {
let size = payload.count
var result = Data([tag])
result.append(contentsOf: [UInt8(0x80 | ((size >> 21) & 0x7f)), UInt8(0x80 | ((size >> 14) & 0x7f)), UInt8(0x80 | ((size >> 7) & 0x7f)), UInt8(size & 0x7f)])
result.append(payload)
return result
}
var decoderConfig = Data([0x20, 0x11, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])
decoderConfig.append(descriptor(0x05, extradata))
var esDescriptor = Data([0, 0, 0])
esDescriptor.append(descriptor(0x04, decoderConfig))
esDescriptor.append(descriptor(0x06, Data([0x02])))
var atom = Data([0, 0, 0, 0])
atom.append(descriptor(0x03, esDescriptor))
return atom
}

In createFormatDescriptionFromMpeg4CodecData:
- atoms.setObject(extradata as NSData, forKey: "esds" as NSString)
+ atoms.setObject(mpeg4EsdsAtom(extradata) as NSData, forKey: "esds" as NSString)

Tests and what is not covered

I added FFMpegVideoDecodingTests to //submodules/MediaPlayer:UniversalMediaPlayerTests. It reads 12-frame 192x108 fixtures through FFMpegFileReader. On master: h264_8bit with hw=true where VideoToolbox can't initialise gives 0 frames; h264_high10, hevc_main10, hevc_main12 and av1_10bit in software give 0 frames each; hevc_main12 with hw=true gives 0; 8->10-bit gives 4 frames and stops at 0.1 s; for mpeg4_part2, VTDecompressionSessionCreate returns -12909. With the fix all of them decode. I'm happy to share the full diff, test file and fixtures if useful.

Not covered:
- VP9 profile 2 (10/12-bit): libvpx is built with --disable-vp9-highbitdepth (third-party/libvpx/build-libvpx-bazel.sh), so it rejects these streams before decoding.
- RGB (gbrp) and monochrome (gray) frames.
- AV1 when the VideoToolbox AV1 decoder rejects a stream (e.g. 4:4:4): the native av1 decoder has no software fallback, so dav1d would be needed.
- MPEG-4 Part 2 in AVI.

Admin DogOct 11

Are the videos working if they are saved inside the Photos app?

Yes. The affected videos play normally after saving them to the Photos app, and only Telegram's in-app player shows the black or frozen picture. That matches the root cause: VideoToolbox can decode these files, but Telegram's FFmpeg-based player path drops the decoded frames (details and fix in the comments above).

ans08 L

Do you have a channel with sample files to share as link?

Yes: https://t.me/tg_video_decode_samples

Each post is one sample file, captioned with what it is and what happens:
- h264_8bit.mp4: control, plays fine
- h264_high10.mp4: H.264 High 10, black
- hevc_main12.mp4: HEVC Main 12, black
- av1_10bit.mp4: AV1 10-bit, black on devices without AV1 hardware (iPhone before 15 Pro, M1/M2 Macs)
- mpeg4_part2.mp4 and resolution.mp4 (real-world file): MPEG-4 Part 2, black
- h264_8bit_then_10bit.mp4: freezes when the stream switches to 10-bit

The two MPEG-4 files were sent as files, because the web client could not preview them.

Did you change the channel username? It's not reachable

Sorry for the confusion — I took the channel down while re-checking, because my first uploads went out as GIFs and documents, so they never reached the video player. It's back: https://t.me/tg_video_decode_samples

Please use the posts starting with "v2" and the last post (mp4v_502x628). They are videos with a 440 Hz tone; to see a frozen picture, watch the orange timestamp burned into the image (the bubble's own countdown keeps running even when the picture is frozen).

Reproduced with Telegram iOS 13.0 (35303) on iPhone 17 Pro Max:
- v2 #3 HEVC Main 12: picture frozen on the first frame, inline and full screen; audio plays.
- Last post, MPEG-4 Part 2 (mp4v, Simple Profile): frozen, audio plays. This is the format of all the real-world videos that fail for me (mp4v Simple Profile written by Lavf57 / FFmpeg 3.4, AAC audio); they play fine in the Photos app. Telegram for macOS freezes on it too.
- v2 #1 H.264, #2 H.264 High 10, #4 HEVC Main 10, #5 AV1 10-bit play fine on this device (hardware decoding).

In an iOS 27 simulator build, current master freezes on v2 #2–#5 and on the MPEG-4 post (the simulator has no hardware decoding, so the software path is used); with the fix from the comments above, all of them play. On real devices the software path is used e.g. for AV1 without an AV1 hardware decoder (iPhone before 15 Pro, M1/M2 Macs) — I haven't verified such a device, and I haven't run the fixed build on a real device.

Correction to my report: H.264 High 10 is not black on iPhone 17 Pro Max.