I've been using emacs for light video editing too, but I took a different approach. Instead of having emacs deal with video directly, I use emacs as a JSON IPC client of mpv, over a unix socket, which handles the video. Over that IPC link I can control mpv playback (optional) and get timestamps/etc to add into an Edit Decision List (EDL) file.