Any tips on running 2x Nvidia 3090s?

RexRed

Any tips on running 2x Nvidia 3090s (non ti)?

I have 1600 power supply, 64gb of ram 3200mhz, 8 cores/16 threads AMD, PCIE 4.0.

Of course, I have my worries this will not work.

I think my wall power has a high amperage, but I am not positive.

I am running two PCs as it is now.

When I am using both computers and run the vacuum and have a space heater on, I never trip the breakers.

My other PC has a 1080ti.

I am wondering about only having 64gb of ram, I thought I read somewhere that i need to have double the amount of system ram than I have for video ram.

Also, I have pcie 4.0 1x16 and one 1x8 lane available.

I am wondering if because the 8 lane pcie is 4.0 if it will be fast enough to utilize the second 3090's potential.

Any thoughts, warnings, suggestions, or advice?

fred9803

This video might have some tips for you.

Gator

Biggest thing is the PSU.  If your 1600 watt PSU is from a good manufacturer, you should be good.  I added a second 3090 a while back, I didn't notice a discernable difference losing some PCI lanes because of the two cards.

RexRed

Awesome info Gator, yea my powersupply is a really good one. 

https://www.newegg.com/rosewill-hercules-1600s-1600w/p/N82E16817182251?Item=N82E16817182251

I currently have 2 x M.2 and 7 of my sata ports used up.

According to Google:

Nvidia 3090 PCIE 4.0 8x results in 5% performance loss.

Nvidia 3090 PCIE 4.0 16x results in 1% performance loss.

PCIE 4.0 is a bottleneck for the 3090.

So it would seem that a 3090TI would provide no performance boost running at PCIE 4.0 16x

It would only gain a larger performance loss.

Ghosty12

One other thing would be is if there was room for two 3090's on a motherboard since they are pretty large cards.

oddbob

My main concern would be cooling it while keeping the noise to a reasonable level, but then I don't like noise.

Gator

oddbob said:

My main concern would be cooling it while keeping the noise to a reasonable level, but then I don't like noise.

Good point.  I went water cooled with the second dual card rig, as the top card lost a significant amount of performance (clock speed) due to the heat off of the lower card.  It was pretty significant even in a full tower.  Sorry no exact figures and memory is a little rusty, but in the neighborhood of 15%.

If you already have 1, you could go with a water cooled AIO for the second card, that would probably help a great deal with good placement of the radiator.

oddbob

Gator said:

oddbob said:

My main concern would be cooling it while keeping the noise to a reasonable level, but then I don't like noise.

Good point.  I went water cooled with the second dual card rig, as the top card lost a significant amount of performance (clock speed) due to the heat off of the lower card.  It was pretty significant even in a full tower.  Sorry no exact figures and memory is a little rusty, but in the neighborhood of 15%.

If you already have 1, you could go with a water cooled AIO for the second card, that would probably help a great deal with good placement of the radiator.

I watercooled my 3090 and it's pleasantly quiet. I've considered a second card but adding another 300-350w would probably mean spinning the fans up past the speed that I want them to be.

The problem with rad placement is that they're either exhausting warm air into the case or pulling warm air from inside the case through them. If I added a second card I'd probably go for an external rad like a Mo-Ra and move the heat and fan noise away from myself and the other parts.

brimstoneomega

Good news about Iray is it's slow enough that bandwidth isn't really your issue. Getting them to actually pool memory is in my experience. I gave up on my 2x 2080 Threadripper system. It could be I was just doing things wrong. Either way, doubling the cudu cores will make a huge difference, so any slowdowns from bandwidth are immediately surpassed by the general massive speed boost to renders, and you don't even need to bridge them for that.

RexRed

Thank you very much for all of the really helpful comments peeps!

I am definitely looking for the larger memory pool as well as more cuda cores.

I often have to pare down a scene because of exceeding the 24gb limit and a project being shifted to my CPU to render. The old blank screen…

As for heat problems, about a month ago I made a hole in my wall that accesses the room on the other side of my computer desk.

Now both of my other computers (desktop and streaming) are in the other room.

So cooling is no problem because I can open the window or even put a large fan directly at the PC.

My computer room is now completely silent.

This by far is the best change I have made to my computer setup in MANY years.

I live in Maine where there are only really about three hot summer months a year.

So I can keep that room relatively cold and so no extra cooling is necessary.

As for fan speed, I could run the fans at high all the time and not hear a bit of noise from the PCs because of the wall between me and the computers.

I record my vocals as well for music my studio is now 100% completely silent.

The computers are only on the other side of the wall, so the cable extension lengths were very minor.

This change is something I have thought about many times and simply disregarded the thought. Mostly because I thought cables would not work. I was wrong.

I got tired of the heat in my studio and the fan noise and the space they take up. None of these are a problem anymore.

As for ram...

I only have 64gb of 3200 system ram.

It is $400-$500 for 128gb of ram.

After just putting out $1400 for the graphics card and another $80 for the NVLINK bridge, I am cooling it on buying anything more for a while.

I cannot find anything on the web about the performance hit I will get when only using 64gb rather than the 128gb recommended.     

 

kwerkx

When I am using both computers and run the vacuum and have a space heater on, I never trip the breakers.

Sorry, I'm a sucker for a good PSA.  You probably already know this; but as they say "winter is coming".  Space heaters are dangerous when on extension cord, or power strip.. but they also generate dirty power that can be detrimental to your electronics.  Link to one case: https://www.fluther.com/210653/what-is-dirty-power-and-is-my-space-heater-generating-it/

Thanks for the question tough, a dual 3090 setup is on my "wishlish".

PerttiA

Electrical power not an issue for those with 240V electricity 

EnoughButter

Good luck! I'm curious about this as well. I have a 3090 and a 1080Ti, but have been too scared to run both, lol, with a 1200w power supply. But with prices dropping for the 3090, I'm debating going the dual 3090s or just getting a 4090, if the benchmarks are favorable. I don't think I need 48GB of pooled VRAM, but doubling my rendering times would be welcome in either case.

RexRed

I reach the 24gb threshold very often.

I nearly always have at least two Gen 8.1 figures in a scene, the largest HDRI I can find, Ultrascenery AND Ultrascenery XT.

Usually an entire environment load of some sort. Hair, beards, clothing and jewelry of all types, some geoshells, a few trees and shrubs, a cat, dog, a few chickens and/or the entire farm, vehicles, water, fog, rain, mist and magic effects and a plethora of VDBs.

I can easily reach 24gb and then I am stuck wondering who or what needs to go (or be hidden) so I can render it... 

RexRed

The second 3090 arrived today and it installed with only one hitch, I bought the wrong NVLINK.

My motherboard has a 3 slot configuration and I bought a 4 slot bridge.

I am going to keep the 4 slot one because I have another tower that has that configuration.

I may switch things around later on.

I don't like that the top card gets hot, sometimes up to 80 degrees.

I figure 4 slots might alleviate that concern some. So might water cooling, but I don't want to disassemble a card to add that.

The 3 slot bridge is "not" found by searching for a 3090 NVLINK, it is found by searching for PNY NVIDIA NVLink Bridge for RTX A6000 (3-Slot)

RTX A6000? (who knew?)

Even without the link both cards render much faster but really that is not a big thing so far to me.

A 3090 really renders fast alone, I mostly want it for the memory pool.

But my Iray viewport is a lot faster too and that part is VERY nice.

With my single 3090, even some of the most complex scenes usually only take a couple hours to render.

Rendering small HD animations might actually be fun to try.

marble

RexRed said:

The second 3090 arrived today and it installed with only one hitch, I bought the wrong NVLINK.

My motherboard has a 3 slot configuration and I bought a 4 slot bridge.

I am going to keep the 4 slot one because I have another tower that has that configuration.

I may switch things around later on.

I don't like that the top card gets hot, sometimes up to 80 degrees.

I figure 4 slots might alleviate that concern some. So might water cooling, but I don't want to disassemble a card to add that.

The 3 slot bridge is "not" found by searching for a 3090 NVLINK, it is found by searching for PNY NVIDIA NVLink Bridge for RTX A6000 (3-Slot)

RTX A6000? (who knew?)

Even without the link both cards render much faster but really that is not a big thing so far to me.

A 3090 really renders fast alone, I mostly want it for the memory pool.

But my Iray viewport is a lot faster too and that part is VERY nice.

With my single 3090, even some of the most complex scenes usually only take a couple hours to render.

Rendering small HD animations might actually be fun to try.

I have a 3090 and I can't imagine having to wait hours for a render. I don't know what kind of complexity you are meaning - maybe you render in very high resolutions? I render at 5:4 ratio (1600x1280) and I have a maximum of four G8 characters in my scenes (I use billboards for background characters). I often use IRay shaders rather than original textures on props and environments because the shaders seem to render quicker. However, I rarely have a scene that takes longer than 5 minutes to render. I complain bitterly about dForce because that is now the feature that slows down my workflow the most.

That said, I have been considering a second 3090 (although I doubt that my 850W PSU or my midi-tower case will suffice) but my reasons are that I would like to get more into animation and waiting 5 minutes per frame is just not going to work for me. It is more likely, however, that I will have to send scenes over to Blender and animate with Eevee although the loss of quality will be an issue. But even Cycles is now faster than IRay.

RexRed

marble said:

RexRed said:

The second 3090 arrived today and it installed with only one hitch, I bought the wrong NVLINK.

My motherboard has a 3 slot configuration and I bought a 4 slot bridge.

I am going to keep the 4 slot one because I have another tower that has that configuration.

I may switch things around later on.

I don't like that the top card gets hot, sometimes up to 80 degrees.

I figure 4 slots might alleviate that concern some. So might water cooling, but I don't want to disassemble a card to add that.

The 3 slot bridge is "not" found by searching for a 3090 NVLINK, it is found by searching for PNY NVIDIA NVLink Bridge for RTX A6000 (3-Slot)

RTX A6000? (who knew?)

Even without the link both cards render much faster but really that is not a big thing so far to me.

A 3090 really renders fast alone, I mostly want it for the memory pool.

But my Iray viewport is a lot faster too and that part is VERY nice.

With my single 3090, even some of the most complex scenes usually only take a couple hours to render.

Rendering small HD animations might actually be fun to try.

I have a 3090 and I can't imagine having to wait hours for a render. I don't know what kind of complexity you are meaning - maybe you render in very high resolutions? I render at 5:4 ratio (1600x1280) and I have a maximum of four G8 characters in my scenes (I use billboards for background characters). I often use IRay shaders rather than original textures on props and environments because the shaders seem to render quicker. However, I rarely have a scene that takes longer than 5 minutes to render. I complain bitterly about dForce because that is now the feature that slows down my workflow the most.

That said, I have been considering a second 3090 (although I doubt that my 850W PSU or my midi-tower case will suffice) but my reasons are that I would like to get more into animation and waiting 5 minutes per frame is just not going to work for me. It is more likely, however, that I will have to send scenes over to Blender and animate with Eevee although the loss of quality will be an issue. But even Cycles is now faster than IRay.

Thanks for your response Marble. Yes, I render at least 3840x2160 sometimes 10,000 x 10,000

I have at least two characters and often have large environments accompanied with ultrascenery and ultrascenery XT, I use the largest HDRIs I can find and the most detailed hair/clothes/skin on my figures.

I also like to use VDBs in my scene whenever possible.

I use multiple spotlights and often have dark areas/shadows that can be pixelated if I do not let it render long enough.

Two hours is often the minimum 2000 iterations. Sometimes it can take four hours and I will let it go up to 6000 iterations or more.

I will use caustic if I want things to reflect off each other (if it will allow me, sometimes I get granular effects and can’t use it) and I engage specular rendering, because I am convinced it can often make my skin textures look more natural.

I set it up to render and go watch a movie while it is going. I do not allow my CPU to participate so I can use my PC in the background if needed.

And no, a 650-watt power supply would probably not work. I suggest a 1600 PS, I have 7 hard drives and 2 M.2 drives I plug in a lot of USB devices like an RME Fireface UCX II etc.  

marble

RexRed said:

marble said:

RexRed said:

The second 3090 arrived today and it installed with only one hitch, I bought the wrong NVLINK.

My motherboard has a 3 slot configuration and I bought a 4 slot bridge.

I am going to keep the 4 slot one because I have another tower that has that configuration.

I may switch things around later on.

I don't like that the top card gets hot, sometimes up to 80 degrees.

I figure 4 slots might alleviate that concern some. So might water cooling, but I don't want to disassemble a card to add that.

The 3 slot bridge is "not" found by searching for a 3090 NVLINK, it is found by searching for PNY NVIDIA NVLink Bridge for RTX A6000 (3-Slot)

RTX A6000? (who knew?)

Even without the link both cards render much faster but really that is not a big thing so far to me.

A 3090 really renders fast alone, I mostly want it for the memory pool.

But my Iray viewport is a lot faster too and that part is VERY nice.

With my single 3090, even some of the most complex scenes usually only take a couple hours to render.

Rendering small HD animations might actually be fun to try.

I have a 3090 and I can't imagine having to wait hours for a render. I don't know what kind of complexity you are meaning - maybe you render in very high resolutions? I render at 5:4 ratio (1600x1280) and I have a maximum of four G8 characters in my scenes (I use billboards for background characters). I often use IRay shaders rather than original textures on props and environments because the shaders seem to render quicker. However, I rarely have a scene that takes longer than 5 minutes to render. I complain bitterly about dForce because that is now the feature that slows down my workflow the most.

That said, I have been considering a second 3090 (although I doubt that my 850W PSU or my midi-tower case will suffice) but my reasons are that I would like to get more into animation and waiting 5 minutes per frame is just not going to work for me. It is more likely, however, that I will have to send scenes over to Blender and animate with Eevee although the loss of quality will be an issue. But even Cycles is now faster than IRay.

Thanks for your response Marble. Yes, I render at least 3840x2160 sometimes 10,000 x 10,000

I have at least two characters and often have large environments accompanied with ultrascenery and ultrascenery XT, I use the largest HDRIs I can find and the most detailed hair/clothes/skin on my figures.

I also like to use VDBs in my scene whenever possible.

I use multiple spotlights and often have dark areas/shadows that can be pixelated if I do not let it render long enough.

Two hours is often the minimum 2000 iterations. Sometimes it can take four hours and I will let it go up to 6000 iterations or more.

I will use caustic if I want things to reflect off each other (if it will allow me, sometimes I get granular effects and can’t use it) and I engage specular rendering, because I am convinced it can often make my skin textures look more natural.

I set it up to render and go watch a movie while it is going. I do not allow my CPU to participate so I can use my PC in the background if needed.

And no, a 650-watt power supply would probably not work. I suggest a 1600 PS, I have 7 hard drives and 2 M.2 drives I plug in a lot of USB devices like an RME Fireface UCX II etc.  

 

Wow! All too rich for my blood ;)

I only have a 3090 because my generous family chipped in to help me replace my 1070 but I was thinking that at the scheduled time for my next replacement I might opt for a second 3090 as the price is coming down rapidly and my current 3090 will not be obsolete in terms of rendering power. As you say though, my 850W PSU will not cope and my PC chassis will need more space, I also have 2 M.2 drives and a couple of SSDs but my final HDD crashed just last week (that's why I have a second M.2).

I make picture stories for my own amusement so I am not going for high quality art. Nevertheless, I try my best to avoid the images looking like typical 3D "comics". I browse through DevArt and am surprised at what some people are happy to put on display there. I would never accept such poor efforts where no consideration is given to photo-realism, lighting, composition, natural posing and body posture and a whole lot more. Even so, I refrain from calling what I do "Art" even though that has brought me into lengthy disagreements with some on this forum. I'm not saying that they are wrong, just that my own idea of art is not something that would include my hobby renders. But generally, my stories have at least 100 images and I like to try for about 10 per day - impossible if I were to wait hours for each. Another problem for me is that I always see something I want to change or improve so in order to get those 10, I probably render at least 30.

Gator

marble said:

RexRed said:

marble said:

RexRed said:

The second 3090 arrived today and it installed with only one hitch, I bought the wrong NVLINK.

My motherboard has a 3 slot configuration and I bought a 4 slot bridge.

I am going to keep the 4 slot one because I have another tower that has that configuration.

I may switch things around later on.

I don't like that the top card gets hot, sometimes up to 80 degrees.

I figure 4 slots might alleviate that concern some. So might water cooling, but I don't want to disassemble a card to add that.

The 3 slot bridge is "not" found by searching for a 3090 NVLINK, it is found by searching for PNY NVIDIA NVLink Bridge for RTX A6000 (3-Slot)

RTX A6000? (who knew?)

Even without the link both cards render much faster but really that is not a big thing so far to me.

A 3090 really renders fast alone, I mostly want it for the memory pool.

But my Iray viewport is a lot faster too and that part is VERY nice.

With my single 3090, even some of the most complex scenes usually only take a couple hours to render.

Rendering small HD animations might actually be fun to try.

I have a 3090 and I can't imagine having to wait hours for a render. I don't know what kind of complexity you are meaning - maybe you render in very high resolutions? I render at 5:4 ratio (1600x1280) and I have a maximum of four G8 characters in my scenes (I use billboards for background characters). I often use IRay shaders rather than original textures on props and environments because the shaders seem to render quicker. However, I rarely have a scene that takes longer than 5 minutes to render. I complain bitterly about dForce because that is now the feature that slows down my workflow the most.

That said, I have been considering a second 3090 (although I doubt that my 850W PSU or my midi-tower case will suffice) but my reasons are that I would like to get more into animation and waiting 5 minutes per frame is just not going to work for me. It is more likely, however, that I will have to send scenes over to Blender and animate with Eevee although the loss of quality will be an issue. But even Cycles is now faster than IRay.

Thanks for your response Marble. Yes, I render at least 3840x2160 sometimes 10,000 x 10,000

I have at least two characters and often have large environments accompanied with ultrascenery and ultrascenery XT, I use the largest HDRIs I can find and the most detailed hair/clothes/skin on my figures.

I also like to use VDBs in my scene whenever possible.

I use multiple spotlights and often have dark areas/shadows that can be pixelated if I do not let it render long enough.

Two hours is often the minimum 2000 iterations. Sometimes it can take four hours and I will let it go up to 6000 iterations or more.

I will use caustic if I want things to reflect off each other (if it will allow me, sometimes I get granular effects and can’t use it) and I engage specular rendering, because I am convinced it can often make my skin textures look more natural.

I set it up to render and go watch a movie while it is going. I do not allow my CPU to participate so I can use my PC in the background if needed.

And no, a 650-watt power supply would probably not work. I suggest a 1600 PS, I have 7 hard drives and 2 M.2 drives I plug in a lot of USB devices like an RME Fireface UCX II etc.  

 

Wow! All too rich for my blood ;)

I only have a 3090 because my generous family chipped in to help me replace my 1070 but I was thinking that at the scheduled time for my next replacement I might opt for a second 3090 as the price is coming down rapidly and my current 3090 will not be obsolete in terms of rendering power. As you say though, my 850W PSU will not cope and my PC chassis will need more space, I also have 2 M.2 drives and a couple of SSDs but my final HDD crashed just last week (that's why I have a second M.2).

I make picture stories for my own amusement so I am not going for high quality art. Nevertheless, I try my best to avoid the images looking like typical 3D "comics". I browse through DevArt and am surprised at what some people are happy to put on display there. I would never accept such poor efforts where no consideration is given to photo-realism, lighting, composition, natural posing and body posture and a whole lot more. Even so, I refrain from calling what I do "Art" even though that has brought me into lengthy disagreements with some on this forum. I'm not saying that they are wrong, just that my own idea of art is not something that would include my hobby renders. But generally, my stories have at least 100 images and I like to try for about 10 per day - impossible if I were to wait hours for each. Another problem for me is that I always see something I want to change or improve so in order to get those 10, I probably render at least 30.

1600 watts for two 3090s is overkill, where some of the high numbers might come from is from meh quality PSUs.  I have 2 3090s, sometimes overclocked and one a FTW3 model, in a 1300 watt PSU.  No issues.  I do have an AMD 5950x, the latest gen Intel are power hogs and will need another 100+ watts.

Plug all your components into a power supply calculator, with 2 3090's and you should be in the 1000 watt neighborhood.  It's a decent idea to over spec 10-20% as the efficiency degrades over time and handling transient spikes.

Stick with a good manufacturer of power supplies like EVGA or Seasonic (I like EVGA as they have stellar customer service IMO) and you should be good.

oddbob

RexRed said:

I don't like that the top card gets hot, sometimes up to 80 degrees.

I figure 4 slots might alleviate that concern some. So might water cooling, but I don't want to disassemble a card to add that.

If 80c is the core temp then the ram may be over 100. My FE card used to really spin up the fans when the ram hit 102. Not sure what temp the card would throttle but it's something to watch out for.

Expense aside watercooling is pretty straight forward now. Corsair waterblocks come with all the pads and paste pre-applied and they have vids on youtube of how to strip and reassemble various cards. Doesn't take long to do.

If it helps, I'm at 52c core and 78c ram during long renders. Single 3090 drawing 340w, +115 core, +500 mem. Low 20s centigrade ambient, water temp high 30s, fans at about 1100rpm.

 

nicstt

Without reading the replies...

Undervolt em!

oddbob

Gator said:

1600 watts for two 3090s is overkill, where some of the high numbers might come from is from meh quality PSUs.  I have 2 3090s, sometimes overclocked and one a FTW3 model, in a 1300 watt PSU.  No issues.  I do have an AMD 5950x, the latest gen Intel are power hogs and will need another 100+ watts.

Plug all your components into a power supply calculator, with 2 3090's and you should be in the 1000 watt neighborhood.  It's a decent idea to over spec 10-20% as the efficiency degrades over time and handling transient spikes.

Stick with a good manufacturer of power supplies like EVGA or Seasonic (I like EVGA as they have stellar customer service IMO) and you should be good.

I've got a PSU with a power display and I'm at about 500w rendering just on the 3090 with the GPU accounting for about 340w. I'm at about 600w while gaming. When I was looking at dual 3090s I was looking at PSUs around the 1200w range.

oddbob

nicstt said:

Without reading the replies...

Undervolt em!

Not a bad call, something like 1850/850 would keep most of the performance and save quite a lot of power draw and heat.

Mine's running at 2100 / 1.1 volt but I've got the cooling overhead for it.

RexRed

Lots of great info here peeps! Thanks for adding all this great input! Lots to consider and think about.

One of my hard drives is an 18tb Seagate drive and one of my M.2 is 4tb (where I keep my Daz content).

I have two 8tb drives another 2tb M.2 and a 12tb drive and a 5tb external drive.

The 1600 power supply only uses the power my devices demand from it.

The 4090s draw even more power.