Advice on setting up a render farm :)

Problems? We'll give um a shot
legmog
BIFF
Posts: 20
Joined: Sun Oct 31, 2010 5:55 pm

Advice on setting up a render farm :)

Postby legmog » Wed Jan 15, 2014 5:39 pm

Hey peeps.
I'm currently looking into building a 6 node render farm and have absolutely NO idea where to start (I must stress hardcore my newbishness at this sort of thing).
Hence I thought I'd do a little post here, as you LAN folks have always been a font of knowledge with computer issues and have helped me out in the past :)

From my research thus far.... I have pretty much surmised that the workflow of creating a render farm would be done in 3 stages...

Stage 1 - Make render nodes
Stage 2 - Fit render nodes into some sort of enclosure (wether all nodes in the same enclosure, or split them up into multiple enclosures)
Stage 3 - Link all the nodes / my main computer together through a network.

Stage 1... I've kind of come to terms with, but stages 2 and 3 are still massive grey areas (maybe grey is too hopefull a term... More like pitch black :p )

I've been discussing this issue on CG Society ( http://forums.cgsociety.org/showthread. ... ost7735214 ). But I figure it's always good to get a second opinion :)
So any words of wisdom are welcome! :)

User avatar
xconspirisist
Pseudo-jolly-fatman, rapid-post-editor, postman-loather, coffee-induced-murderer, meatball-fanatic, GPS-hater, vapourware-author, executioner-of-flying-bugs, eternal emperor of entirety
Posts: 6182
Joined: Tue Sep 07, 2004 1:04 pm
Battle.net: xcons#2832
steamid: xconspirisist
Location: Hyrule. ▲
Contact:

Re: Advice on setting up a render farm :)

Postby xconspirisist » Wed Jan 15, 2014 7:11 pm

So, this is quite a substancial portion of my job; advising people on how to build architectures like these. The largest project I've personally advised for is around about 350 nodes (servers), so I've got a little bit of experience. The company I work for (Red Hat) has built a render farm that both Dreamworks and Pixar use. :) Now that that bit of willy waggling is out of the way, the advice that follows is as unbaised, and is based on my personal experience. But as always, you should seek as many opinions as possible.

The first thing you should should determine, is if you are building a grid or a multi-node application environment (aka cluster, distributed application, supercomputer). The difference between the two is simple; grid have "workers" that take a small amount of input (job), run for a while, and produce some output. For a render farm, the rendering of your video/scene is split into jobs like: "render 10 scenes of this", "render 10 scenes of that", etc. I'll ignore the second option for now, as this is almost certainly not what you want.

The next step, would to be to determine what rendering software you have at the moment and what rendering software you have that is capable of working in a render farm. Most vendors for products like 3DSM, Maya, etc all have "renderers" available that can run on a minimum installation of an operating system. This is a great start.

Once you've chosen the renderer you're going to use, you need to find a grid scheduler that is compatible with it. A grid scheduler is the bit of software that takes your one giant task (video/scene) and splits it up into lots of little workable pieces. The scheduler sends those pieces to compute nodes, which run and then send their result back to the main server or store it on shared storage. Just a note on terminology, in clusters/farms/whatever, the servers are called "nodes". A convention is that "compute nodes" are the nodes that do the computing (or in your case, rendering), and you have many of them. In each cluster there is a "head node", which runs the scheduling software.

Then, you need to setup a cluster of "compute nodes", a common installation that contains the renderer you are going to use, as well as some utilities, like the scheduling client. It's normal to setup a "provisioning system" on your head node, that is used to control and re-install compute nodes very quickly and easily when you need to. The head node and compute node should all share the same network, normally.

Hopefully you find this information conceptually useful, now for the technology.

One of the biggest mistakes you can do with a render farm, or indeed any sort of cluster, is to buy hardware first, then work out the software you're going to use/workload, second. You should determine if your renderer wants, or needs special instruction set architectures like AVX2 (recently came in Haswell), or if your workload is more paralell, could you make use of a CUDA compatible graphics card? I've seen plenty of Universities build clusters out of spanking new i7's, when a couple of CUDA graphics cards and a bit of C++ would rip through their workload at half the cost in half the time. You actually can't spent enough time doing research.

Finally, I've not mentioned any software yet, but I'll just declare my allegiances - my advice so far has been as non-baised as I can make it; I work for Red Hat, I'd advocate a Linux environment because that is what I know (and I know it works!). You can probably do all this with free software, the OS could be CentOS, the renderer could be Condor, but until you answer the questions about renders and that sort of thing, you're wayyyy off making those sort of technology decisions!

legmog
BIFF
Posts: 20
Joined: Sun Oct 31, 2010 5:55 pm

Re: Advice on setting up a render farm :)

Postby legmog » Wed Jan 15, 2014 11:04 pm

Yoooo xconspirisist
Thanks for the detailed response!

I'll address your points...
1st point - As to grid / multi-node application environment. Which ever one of those 2 things is the one that SPLITS the job.... THAT'S the one ^^
I'm planning on doing 6 nodes.... So say I have a 60 frame animation, each node would render 1/6th of the total frames (10 frames in this case).

2nd point - What rendering software...
So I use Cinema 4D, which comes with a network render software called 'Net Render'.
You can view a description of it here http://www.youtube.com/watch?v=09hP2XAi0Rk (it's a 20 minute video, but first first 70 seconds explain it quite well).

3rd point - As far as I can tell, I believe 'Net Render' also works as a 'grid scheduler' of sorts.

4th point 'cluster of 'compute nodes' - I guess I'm a little fuzzy on terminology here, so the following may be totally unrelated to what you ACTUALLY meant... But here goes :)
My planned setup was going to be my main workstation computer. Then the 6 nodes wacked in some enclosure.... Would THIS constitute the 'cluster of compute nodes'? And would the 'head node' be my main computer which I work from?

5th point (figuring out software before hardware) - Well I know I will be using Cinema 4D, and the 'Net Render' software (both C4D and Net Render are made by MAXON and come together as a bundle). I know (at least as far as C4D goes), the most important factor in render speeds / efficiency is the processor. Graphics Cards don;t come into play when rendering, and I've been told to not even bother implementing Graphics Cards into the nodes on the CG Society forum.

Finally, as to software environments and such like ^^
I've honestly given no thought to the OS that would be installed on these Nodes...
My current workstation is Windows 7 Ultimate, and I guess I was blindly assuming I'd install the same onto the nodes (or some such similar Windows 7 based OS that can handle 64bit).

User avatar
Merson
Ban Hammer Admin
Posts: 1893
Joined: Mon Oct 22, 2007 10:26 pm
Origin ID: Merson
Battle.net: Merson#2706
steamid: Merson
Location: Galway, Ireland
Contact:

Re: Advice on setting up a render farm :)

Postby Merson » Wed Jan 15, 2014 11:09 pm

legmog wrote:Graphics Cards don;t come into play when rendering, and I've been told to not even bother implementing Graphics Cards into the nodes on the CG Society forum.

Sure about that? :)

http://www.m4d.info/index.php?id=15336
http://www.nvidia.com/object/nvidia-iray.html

Jani
Swearing Admin
Posts: 1946
Joined: Thu Sep 08, 2005 1:23 pm
Location: Plymouth
Contact:

Re: Advice on setting up a render farm :)

Postby Jani » Thu Jan 16, 2014 5:33 am

I know this doesn't help but once it's built can you render some video for me? Got about 500gb of TV episodes I need doing :(

User avatar
Freelance
1337 Krew
Posts: 1610
Joined: Tue Dec 30, 2003 1:35 pm
Origin ID: scratched_uk
Battle.net: scratched#2526
steamid: scratched

Re: Advice on setting up a render farm :)

Postby Freelance » Thu Jan 16, 2014 11:24 am

Jani wrote:I know this doesn't help but once it's built can you render some video for me? Got about 500gb of TV episodes I need doing :(

It's probably not what you meant, but it can be important. When you're spending a decent chunk of change on hardware to keep it occupied doing something so you're getting value for money, rather than sitting idle.
I adblock smilies.

User avatar
xconspirisist
Pseudo-jolly-fatman, rapid-post-editor, postman-loather, coffee-induced-murderer, meatball-fanatic, GPS-hater, vapourware-author, executioner-of-flying-bugs, eternal emperor of entirety
Posts: 6182
Joined: Tue Sep 07, 2004 1:04 pm
Battle.net: xcons#2832
steamid: xconspirisist
Location: Hyrule. ▲
Contact:

Re: Advice on setting up a render farm :)

Postby xconspirisist » Thu Jan 16, 2014 11:15 pm

Yoooo xconspirisist
Thanks for the detailed response!


Pleasure ;)

I'll address your points...
1st point - As to grid / multi-node application environment. Which ever one of those 2 things is the one that SPLITS the job.... THAT'S the one ^^


Well, that is how a grid works, but you need grid-like software to do this for you.

I'm planning on doing 6 nodes.... So say I have a 60 frame animation, each node would render 1/6th of the total frames (10 frames in this case).


Sounds good.

2nd point - What rendering software...
So I use Cinema 4D, which comes with a network render software called 'Net Render'.
You can view a description of it here http://www.youtube.com/watch?v=09hP2XAi0Rk (it's a 20 minute video, but first first 70 seconds explain it quite well).


This looks exactly like what I was asking - solves the problem for you.

3rd point - As far as I can tell, I believe 'Net Render' also works as a 'grid scheduler' of sorts.


Agreed.

4th point 'cluster of 'compute nodes' - I guess I'm a little fuzzy on terminology here, so the following may be totally unrelated to what you ACTUALLY meant... But here goes :)
My planned setup was going to be my main workstation computer. Then the 6 nodes wacked in some enclosure.... Would THIS constitute the 'cluster of compute nodes'? And would the 'head node' be my main computer which I work from?


A compute node does what is is told to do (runs some sort of net rendering software, in your case) and you have 6 of them. A group of machines doing the same thing is called a cluster. The head node sends out commands - this would be your main computer that you work from.

5th point (figuring out software before hardware) - Well I know I will be using Cinema 4D, and the 'Net Render' software (both C4D and Net Render are made by MAXON and come together as a bundle). I know (at least as far as C4D goes), the most important factor in render speeds / efficiency is the processor. Graphics Cards don;t come into play when rendering, and I've been told to not even bother implementing Graphics Cards into the nodes on the CG Society forum.


You could well be talking rubbish - check out Merson's post, and research into the Graphics card support for NetRenderer. My pure guess is that if NetRenderer support CUDA, then you could cut your number of compute nodes down to 2, and invest in good graphics cards.

Finally, as to software environments and such like ^^
I've honestly given no thought to the OS that would be installed on these Nodes...
My current workstation is Windows 7 Ultimate, and I guess I was blindly assuming I'd install the same onto the nodes (or some such similar Windows 7 based OS that can handle 64bit).


Looks like NetRenderer runs on Windows, so that decision is made for you. You know how to use windows too, so that is easy ;)

legmog
BIFF
Posts: 20
Joined: Sun Oct 31, 2010 5:55 pm

Re: Advice on setting up a render farm :)

Postby legmog » Tue Jan 21, 2014 2:01 am

Hi people
Sorry for the late response.... Am inundated with a shit ton of work right now >.<
Firstly, as to the issue of the whole GPU vs CPU thing.

I DO like the look of iRay, but don't know wether I'm totally sold (the other NVIDIA site was not compatable with C4D).
If C4D had a natural in-built option for GPU rendering, I would seriously consider this an option (it however, does not). Thus I'm hesitant to use third party plugins, mainly on account of 'if it's not broken, don't fix it'.
I've got several 'balls against the wall' jobs on the go at once right now, and several lined up in the future (which would be requiring a render farm), and I just don't have the time to get accustomed to a whole new rendering application. ''It's not the tools you have, it's how you use them'' as they say :) and I've got 9+ years of expertise and experience in using the C4D renderer.
I'm fully experienced and understand the way it works, and how to get great looking visuals from it. I don't want to jump head first in the deep end with software I have zero experience with, during a prolonged period of very demanding jobs.

That's my main qualms with GPU.... I will however download a trial and give it a spin. See if it can sway me, but right now I'd rather stick with CPU.

Trying to figure out just what this render farm will comprise of is proving a bit of a challenge, as it seems for every question answered, another 10 pop up >.<
But this is what I've come up with thus far...
For software, I'm set on C4D Net Render, with Windows 7 OS.

Hardware...
- Intel Core i7 4930k 3.40GHZ Hex Core
http://www.ebuyer.com/538161-intel-...-bx80633i74930k

- 16GB DDR3 RAM, Crucial
http://www.ebuyer.com/309547-crucia...kit102464ba1339

-HDD Scorpio BLK 250GB
http://www.ebuyer.com/523951-hdd-sc...16mb-wd2500bekx

- Coolermaster Elite Power 500W PSU
http://www.ebuyer.com/253900-cooler...rs500-psapj3-it

-Cooler Master XDream i117 Fan
http://www.amazon.co.uk/Cooler-Mast...m/dp/B005C5GNQS

For enclosures, I was looking at a case like this for each node...
http://www.ebuyer.com/390900-caseco...49-case-kk-9949

I guess my main issue is what kind of MOBO could I use with this aforementioned setup?
I posted one on another forum and they told me it would be incompatable ( http://www.ebuyer.com/429725-asrock...rboard-h61m-dgs )

Asides from that, what is your guys thoughts on the list thus far?



JANI - Sure :D
If / when this render farm gets done, and it's sitting idle, send me some stuff over :)
What is it you're working on?

User avatar
Houlden
Unter Peon
Posts: 192
Joined: Thu Jan 05, 2012 1:30 am
Location: 45-58-32

Re: Advice on setting up a render farm :)

Postby Houlden » Tue Jan 21, 2014 10:42 am

I won't pretend to know anything about rendering farms whatsoever, but you'll need a 2011 socket motherboard for that processor, which are a litttttle bit pricier than I guess you would've liked to spend (based on how much that Asrock board was), but yeah, here's a link:

http://www.scan.co.uk/shop/computer-har ... otherboard

Make sure that board you're buying has had it's BIOS updated to support Ivybridge-E, most if not all should have been by now, but even so you don't want to get your nice expensive board and it doesn't work.
i5 3570k - Hyper 212 Evo - ASRock Fatal1ty Z77 Performance - 2x SLI MSI 760 - 128GB Samsung 830 - 1TB WD Red

User avatar
Merson
Ban Hammer Admin
Posts: 1893
Joined: Mon Oct 22, 2007 10:26 pm
Origin ID: Merson
Battle.net: Merson#2706
steamid: Merson
Location: Galway, Ireland
Contact:

Re: Advice on setting up a render farm :)

Postby Merson » Tue Jan 21, 2014 2:20 pm

2011 socket also uses quad channel memory so ideally you would want 4x4GB sticks. See my CPU-Z validation below as I have a 2011 setup myself.


Return to “Tech Advice”