Serato Video General Discussion

Forum
 Serato Video General Discussion
Video SL Encoding Farm
Talk about Serato Video and Video-SL.

Video SL Encoding Farm

nik39 12:33 AM - 22 July, 2008
Last week I decided to re-encode my video library. Unfortunately I was too quick with my actions which had the consequence that I deleted my old encoded videos. Since I had a gig on saturday and I started late on friday I really had to hurry up. To speed up things I took all my machines which had at least a little bit of power and decided to let them help encoding the videos.

Pics:
urban-breaks.de <- click
urban-breaks.de <- click

These were the machines involved:
1x MacBook Pro, C2D Dual Core 2.16GHz, 2GB RAM, Win XP
1x Dell 9300, Centrino Single Core 2.16GHz, 2GB RAM, Win XP
1x Dell D830, C2D Dual Core 2.2GHz, 4GB RAM, Win XP
1x Sony Vaio, Pentium Mobile Single Core 1.2GHz, 128MB RAM, Win 2K
1x Dell Vistro 1500, C2D Dual Core 2GHz, 3GB RAM, Win XP
1x FujitsuSiemens, Dual Xeon Quad Core 2GHz, 1GB RAM, W2K3 Server
1x Custom Made, Athlon Single Core 1.6GHz, 768MB RAM, Win XP
1x Custom Made, C2D Dual Core 2.4GHz, W2K Server
1x Custom Made, C2D Quad Core 2.2GHz, 8GB RAM, Ubuntu/Debian

1x FujitsuSiemens external USB drive (source material)
1x Maxtor USB drive (encoded video storage)

All networked by two GBit switches (two due physical reasons, one would had been sufficient).

That makes a total of 19 cores, 2 of these I couldn't use as they were on a server which was needed to work properly, 1 of them crapped out on a reg basis (I suspect faulty RAM). It was quite easy to get them all connected and running. On the Windows machines I installed cygwin (a unix like environment) and on the Linux Ubuntu machine I used two virtual machines which were running XP (also with cygwin installed), same goes for the MacBook pro. It ran a VM using Fusion from VMWare with XP inside (reason for not using my bash scripts natively was that I haven't found a good ffmpeg compile on OSX yet). Cygwin was needed cause I coded a few scripts in bash (using the unix environment) which took care of spreading the files, encoding them, putting them back to the master server etc. As the encoder I used ffmpeg. Target format was mp4 as a container, xvid at 2500kbps as the video codec and aac at 160kbps as audio codec. The results are pretty good I would say (I am not an expert here) and the videos run flawlessly with VSL.

I had a throughput of something like 120+ files an hour, not bad :) To make sure that as little stress as possible was put to the machines I made sure that they had virtual ramdisks (to minimize concurrent harddisk access to speed up things). Also I used a big fan to make sure they are as cool as possible.
DJ Czar 5:19 AM - 22 July, 2008
Nik, this is very, very impressive! Perhaps you could share some of these scripts, although I doubt many would be able to implement them in their own environment (even I myself, a video professional, am not much of a programmer.) May I ask why you're encoding with the xvid codec via ffmpeg and not h.264 via x264? I've recently done many tests via x264 and the results are amazing! Also, what are your inputs and are you encoding at different resolutions to compensate for various AR changes or keeping everything at one resolution with letterboxing in some videos?
Freedom 9:46 PM - 22 July, 2008
Nik39 props... thats a good amount 120+ an hour and good conver.....

Im digging that pic reminds me of my house
KMXE 10:32 PM - 22 July, 2008
Quote:
(I am not an expert here)


haha - i'd say you were close to being an expert mate! amazing stuff!

i smell a business opportunity here

"I'll convert your entire video collection for a fee"
Charlie Five 10:42 PM - 22 July, 2008
NICE...

I get about 15 done an hour (thats at single pass). Macbook Pro 2.5, 4 gb ram. I have 2 so make it 30.
RaveDave 11:56 PM - 22 July, 2008
Rofl. Thats awesome man. You have way too many laptops though man...
nik39 12:31 PM - 23 July, 2008
Quote:
I get about 15 done an hour (thats at single pass).

Hehe, forgot to mention... I am converting them in dual pass ;)


Quote:
You have way too many laptops though man...

Well, one of them is not mine... and actually I only have one recent laptop. Thats the Dell D830. The MBP is 1 1/2 yrs old. The other Dell is 3 yrs, that Sony thing is ancient I guess ;)


Quote:
Perhaps you could share some of these scripts, although I doubt many would be able to implement them in their own environment (even I myself, a video professional, am not much of a programmer.)

I think it is really easy to implement them *but* after the experience I gained with ScratchTools people hate command line interfaces. I must agree that ScratchTools is also not very easy to deal with at the beginning and after some time of not using ST I catch myself looking into the docs to find out the proper comand syntax. Plus... well the scripts work fine for me - but they are sort of ugly. I thought about releasing it, but I guess I will be wasting time in supporting it, writing documentations etc. thats not worth it, because...

I am planing to code a GUI frontend for a tool which will be able to rip your Promo Only and MixMash DVDs, name them automatically and convert them into a proper VSL format... ideally it should also support encoding on multiple machines.

BTW: Not really looking forward into VOB support since VOB's do not support tagging, while mp4 does.


Quote:
ay I ask why you're encoding with the xvid codec via ffmpeg and not h.264 via x264?

H.264 puts too much load on my current MBP. That's the only reason. But... with the current process it's just changing one line in the code to re-encode everything in H.264. That is something for the future (can't wait for the quad core laptops :) ).


Quote:
Also, what are your inputs and are you encoding at different resolutions to compensate for various AR changes or keeping everything at one resolution with letterboxing in some videos?

Since VSL has to adjust the videos anyway I'll leave that totally to VSL. Right now I am just de-interlacing the videos but everything else like size, resolution, framerate is kept. I don't see a logical reason to alter these changes (and I am conforming here to a-swift's (his encoding scripts look awesome by the way :) ) suggestions).


I'll keep you posted.
DJ Czar 2:05 PM - 23 July, 2008
Quote:
Since VSL has to adjust the videos anyway I'll leave that totally to VSL. Right now I am just de-interlacing the videos but everything else like size, resolution, framerate is kept. I don't see a logical reason to alter these changes (and I am conforming here to a-swift's (his encoding scripts look awesome by the way :) ) suggestions).
eep you posted.

Well that's the thing with AR changes, if your output is a widescreen display (and most venues nowadays have just that), and you give VSL a letterbox 4:3 video, it will create an unwanted pillarbox effect. It has no way of knowing that you are feeding it widescreen content inside a 4:3 frame. Would be nice if you could tag each video with cropping information, but if you're transcoding anyway, I don't see why you wouldn't want to remove the letterboxing in the first place... It's only healthy..
DJ Czar 2:09 PM - 23 July, 2008
Also, VOBs use non square pixels, but when VSL plays back a 720x480 resolution video, it does so using square pixels, so the original aspect ratio is lost. Again, would be nice to tag each video with pixel aspect ratio info, but if you're transcoding, you can easily resize the video to a square pixel resolution, keeping vertical resolution the same (ie 720x480 becomes 640x480, the 480 is maintained).
nik39 2:32 PM - 23 July, 2008
Warning: As said before, I am no expert...

Quote:
Well that's the thing with AR changes, if your output is a widescreen display (and most venues nowadays have just that), and you give VSL a letterbox 4:3 video, it will create an unwanted pillarbox effect. It has no way of knowing that you are feeding it widescreen content inside a 4:3 frame. Would be nice if you could tag each video with cropping information, but if you're transcoding anyway

Changing formats will always affect the quality, I think we agree with this. Since VSL will scale the videos anway why should I worry and add another format/AR changing step?

I haven't recognized that VSL produces a pillarbox effect, I thought it would crop the video accordingly but I might be wrong.

Also I haven't seen VSL effing up the AR's.

An additional step (changing resolution and 'fixing' the AR)... what would it do that VSL does not do already?


Quote:
Also, VOBs use non square pixels

I thought VOB is a generic format not telling you which resolution you have to use, therefore you can not say that it always uses non square pixels. Wrong?


The encoding process is pretty flexible. Right now I supply the artist, songname, year and source as a part of the filename, the encoding process adds all this info as tags to the mp4. So it should be easy to add identifiers to the filename to distinguish different AR's. But.. as previously asked - right now I don't see a benefit yet. But I am open for discussion :)
D-Twizzle 3:57 PM - 23 July, 2008
very nice setup. my process is very slow compared to this.

i agree with dj czar about needing some type of tag that goes on the individual video that tells vsl specific info about how to handle the aspect ratio and letterboxing.
nik39 5:10 PM - 23 July, 2008
Quote:
very nice setup. my process is very slow compared to this.

I am actually lucky that two of those servers are not in use right now. Otherwise it would take a lot longer to encode.

But hey, in the end... if it takes two weeks, I don't mind, as long as I don't have to babysit the computers.
Freedom 8:10 PM - 23 July, 2008
Nik39 - when or if you ever work out that gui let me know. :)

Im a amazed at the shear knowlege you put into furthering Serato, keep it up
nik39 9:19 PM - 23 July, 2008
Thanks man... :) I'll keep ya posted.
Dj Nyce 7:57 PM - 3 August, 2008
nik what kind of throughput are you getting on the vm's? are you giving 1 core to each vm or are you sharing all resources across the vm's?

i would think that you would get less throughput in the vm guest versus doing the encoding natively on the server.
nik39 11:34 AM - 4 August, 2008
Quote:
i would think that you would get less throughput in the vm guest versus doing the encoding natively on the server.

+1

I did not measure it, but the performance was definitly okay.

I try to give as much sources to a single VM as possible. So the dual cores gave two cores to a single VM, same goes for the quad cores.. two cores to a single VM, meaning I ran two VMs on the quad core.
nik39 11:34 AM - 4 August, 2008
Just wanted to add... that since I didn't want to configure everything from scratch on each VM (like RAMdisk etc.) VM's were the way to go.
Dj Nyce 10:51 PM - 4 August, 2008
have you tried vmware esxi? it's basically vmware server without the host os. it allows you run vm's natively in the esx kernal (without an operating system).
nik39 11:02 PM - 4 August, 2008
The point is that I want to use existing hardware which is in use but can be used for a couple of days. I don't want to install ESXI on it. The less changes I have to do - the better. And (usually if I have enough time) I don't mind if the encoding process is about 10% longer or something (I guess that would be the slowdown on current hardware which supports virtualization). :)
Dj Nyce 1:51 PM - 5 August, 2008
i just started using esxi and i'm noticing a huge performance on my vm's.

anyways is there any reason that you aren't using the ffmpeg win32 builds? i don't have a problem encoding to xvid, x264 or mpeg4.

i'm also curios about the settings that you're using such as gop size, b-frames, b-frames strategy and trellis.

thx
nik39 2:02 PM - 5 August, 2008
Quote:
i just started using esxi and i'm noticing a huge performance on my vm's.

As said... the problem is that I want to apply as little changes as possible to the existing computers. I don't want to install another OS! If you have dedicated machines... esxi would probably be the way to go... (Though I must say in such a case I would install linux natively and run the encoding all natively in linux)

Quote:
anyways is there any reason that you aren't using the ffmpeg win32 builds?

What do you mean? I am using the win32 ffmpeg binaries. :)

I just dont have a good OSX ffmpeg binary.


Quote:
i'm also curios about the settings that you're using such as gop size, b-frames, b-frames strategy and trellis.

Uh... I decided not to deal with these stuff - too complicated. Have you ever checked the man(ual)pag of ffmpeg?? Incredible.

I leave everything as it is, except for the keyframe setting and maximum bitrate.
nik39 1:05 AM - 17 December, 2008
CZar, 101 on AR is needed :)

BTW: Big ups to CZar - the king of video-knowledge on this board.
Dj.Mojo 11:36 PM - 11 November, 2009
BUMP

Any updates on "THE SCRIPT"?

Loading and cueing every single file in handbrake is a pain in the ass!
nik39 11:46 PM - 11 November, 2009
The "script" is not the problem... The tool with the GUI is the problem.. :-/
Dj.Mojo 8:27 AM - 12 November, 2009
Who needs a GUI if he has got a Konix Tutorial? :-D
misky 3:16 PM - 15 November, 2009
If the batch script can be used already, could you please publish it?
Thanks!
nik39 5:08 PM - 15 November, 2009
If you want to figure out how to use it... sure. I didn't write a manual.
DJ Dan-E 5:38 AM - 24 November, 2009
Nik - out of curriosity how many files did you encode? I just spent about 2.5 months converting a little over 10K Videos to H.264 dual pass 3200/160acc However I didnt have scripts used Sorensen Squeeze on (2) Quad Core I7 920 pcs running 64bit os.
nik39 8:07 AM - 24 November, 2009
That's an average of 5.5 videos per hour, I think I had a throughput of 120+ per hour.
djpuma_gemini 6:34 PM - 9 February, 2011
bump...
just playing
nik39 8:01 PM - 9 February, 2011
Ha, this looks sooo unprofessional.

Wait for the 2011 edition ;)
djpuma_gemini 8:20 PM - 9 February, 2011
haha. I was doing a search for encoding scripts and found this thread.
nik39 9:37 PM - 9 February, 2011
Encoding scripts are so 2000 and late ;)
misky 3:53 PM - 21 February, 2011
give it to me :-D