Successfully eGPU'd a Tesla M40
Daz 3D Forums > General > The Commons>Successfully eGPU'd a Tesla M40
Successfully eGPU'd a Tesla M40

in The Commons
Hi all,
Just to let you know, that it's possible to eGPU a Nvidia Tesla M40 24GB through a regular USB keyed riser card (as used in miners) for cheap...
(I often had renders which exceeded my dedicated 8GB Quadro M4000)
Setup is not sexy (yet) and very very noisy but it's about 5x faster than with CPU only...
I can post details if you're interested...
Cheers & happy rendering!
Peter


Screenshot_20200426_202906_com.android.gallery3d.jpg
1080 x 2340 - 2M


Screenshot_20200426_202845_com.huawei.himovie.overseas.jpg
1080 x 2340 - 1M
Comments
Wow - cool! Thanks for sharing this, opens up a lot of possibilities.
Used M40 24 gb are about half the price or less of a used M6000 24 gb. The M6000 is a Workstation graphics card with a variety of video outputs, while the M40 is an "accelerator card" with no video outputs. But both appear to have very similar performance (please correct me if any of this is wrong).
Details would be appreciated!
Wow - a top-rated seller on EBAY is now selling these for $228 dollars with free shipping! The cheapest used M6000 24GB card I have seen was $850, and most are a lot more than that.
Be interested to know which external blower you added for cooling and where you found it. That is one downside to this card - you need a big case if you want to mount internally with the blower.
...have the case.
The downside though, only passive cooling (no fan) so you would need to rig up some form of external cooling if mounted internally (better response). You'd need a to install third party cooler (which adds to the length of the card which is already 10.5")..Unable to find any water cooling solutions for the M40.
So many questions.
Why use a USB connection rather than just a standard PCIE riser? With a PCIE riser you'd get the PCIE power rather than needing to get that 75W from somewhere else. Miners use PCIE risers BTW.
Why add the blower at all? Those cards come without fans because they don't get that hot being out of the case should give it better cooling since it isn't near other hot components and the room airflow helps.. At worst some sort of desk fan should be enough to cool it and it would be a lot quieter than a blower.
While I'm sure 24Gb of VRAM is nice only a 5x speed increase over some old Intel CPU is not that big a jump. Unless you really need the VRAM seems to me you'd get a much better performance bump from a 2060 Super which would cost roughly the same, or less, than the M40.
...the downside of an external GPU is the need to keep the case open for the connections which sort of defeats the design for positive air flow, and leaves it open to dust contamination. Many, if not most cases today have washable filters on the intake fans, even my 7 going on 8 year old P-193 .
If you create big scenes, that 24 GB becomes a speed factor in and of itself as it greatly reduces the chance of the process dumping to much slower CPU.mode (and possibly even slower swap mode if internal memory is exceeded). The change to a 12 GB Titan X alone has made a significant improvement for much of my work..
You must be careful on ebay as the card for too cheap is often not really the card you wanted to buy. >_>
..I'm finding M40s for between 315$ and 450$. As always, on eBay you need to check the seller's rep.
The M40's are designed for special server cases that have a very high airflow through a very limited space, so calling them "passive" coolers is misleading. They come with mounting holes for blowers next to the power plugs. There is at least one YouTube video discussing cooling for the Tesla series. On another forum, several folks reported that the Tesla series overheated immediately in a standard case with standard airflow.
I found a vendor on ebay that is selling a special shroud and fan mount that takes standard 60 mm fans, for $10 plus shipping (it takes the place of the bracket). The alternative ones that mount the blowers come from Australia, and with shipping are pretty expensive.
The M40 has similar specs, same GPU, etc. as the Titan X Maxwell (far as I can tell), but there is a 24GB model vs. 12GB max for the Titan. I have been pretty happy with the boost I get from my used Titan X, so I decided to try for one of the cheap M40 24GB.
The seller is "Top Rated" with over 11,000 sales, and offers a 30-day warranty with free returns. Figured worth a shot. At the price I got, I could get 3-4 of these for the current price of a used 24GB M6000, with similar performance! There is no video output at all on the M40, but that does not matter in this case.
These particular units come with a bracket for a blower, but no blower shroud. Also ordered one of the shrouds with the 60mm fan adapter, it looks like it is shorter than the bracket plus blower. The server board (Supermicro MBD-X9DR3 with 2x E2680V2 CPUs, EEB form factor) I want to use this with is in a huge EEB form factor case, but I still have doubts it will fit without taking out a drive cage. Although the card is the same length as the Titan X, the addition of cooling adds to the overall length. I have a PCIE x16 riser card with a fairly long ribbon cable, so I may need that.
I will report back when/if I get this running.
Looking forward to more info and comments from Peter (thanks again for bringing up this topic, and showing us the M40 will work with IRAY!)
Oh, buybest16 is still selling them for $228.98 on Ebay. They are not tagged with 24GB as a search term, you have to look in the description and at the images of the card.
Manufacturer's part code also matches the 24GB model.
Generic GPU (or CPU) blocks can be put on basically anything with a little less than elegant affixing (though GPU blocks tend to have the holes on the side, so are preferable).
I've seen zip ties used..
I have heard that coolers for the GTX 970/980/TitanX will work on the M40 as well. One of the EVGA HYBRID coolers (water cooling for the GPU, air for the memory) would be interesting to try. Going to try the end-mounted blower first, then we will see. Looks like uezi/Peter used a blower and a shroud/adapter (from the picture).
Very true! I generally look for a top-rated seller with lots of sales and free returns. Had pretty good luck so far.
Generally GPU layouts don't really vary a whole lot, so it's a distinct possibility. If you can find some bare card photos to compare you can probably be a bit more sure.
Air cooling of memory is usually more than fine, on lots of video cards it's completely naked (with lots exposed on the bottom) without causing problems.
Correct. The M40 is NOT A PASSIVE gpu. It was designed for high performance computing and is meant to reside in a server with fans for the card and in cooled server room.
I monitored the temperature via GPU-Z and it was ok (~47 °C).
Mine came from Australia... most likely the same seller... But I waited for weeks...
I think I saw a video on YouTube where someone built a fan with by funneling a "regular pc fan" with some cd-case plastic & duct tape... Another vid showed a guy who removed the casing from the Tesla and mounted a fan onto it...
I got mine (roughly a half year or year ago) for about 500$ on eBay.
If your case has a large enough space to mount the GPU, fan and has enough juice to power it, you could also mount it inside...
I had neither the power (Fujitsu non-standard psu so I couldn't just upgrade it) nor the space (it would have required me to remount a fan). I'll look into it if I can still force it in somehow...
The mining riser I used is only x1 speed so it took a while for loading the scene (Iteration 1 started at 61s) but then milling out ~150 Iterations in 5 minutes... I had reached the limit of 5000 Iterations after 9834s while the CPU one stopped at 03751 iterations after 40005s.
I went with a mining riser as I believed it would be more galvanically separated when adding an additional PSU for the GPU... (plus I always wanted to have one of these).
Another nice side-effect with the riser is that I can pull remove the setup by unplugging the GPU-sided USB... (ATTENTION: MAKE SURE EVERYTHING IS TURNED OFF !!!). Computer still starts and has no apparent issue with it... So only during heavy rendering times, I'll setup the Tesla... normal days will be with my M4000 ;-)
Looking forward to see your results ! (Please make sure that you're feeling safe or test the setup on another old PC ()id did this)
My steps:
The link which gave me confidence (in dual PSU-ing):
https://www.biostar.com.tw/app/en/event/crypto_mining/page4_2.htm
An arduino controlled fan (directly mounted on the GPU):
https://github.com/rgrosset/CoolingTeslaK80
I work with servers. Tesla cards should not overheat even without the sort of airflow most server cases have, also we don't chill the room air below 70F. There was a time when that was done but it greatly increased the expense of operating the datacenter. Now we concentrate on having cases with lots of airflow or ones that don't generate that much heat (we have solid state storage devices that have 1 intake and 1 exhaust fan, both are standard PC 80mm fans not the Delta's we use everywhere else.).
For Tesla's they don't just go in servers with no fans. I've seen them in workstations still without a fan. The one in question is rated for 250W which it pretty much cannot hit, that's the same TDP as the equivalent consumer card and the Tesla is downclocked and never drives video. If it is overheating outside a case then there is an issue with the card. Not surprising with cards sold used since they would have done to 2-4 years of 24/7 server duty before winding up in the used market.
The K40 and M40 Tesla models I have been looking at have a shround which is only open at the ends, so air flow is very limited. I believe that if you just removed the shroud, they would do just fine in a well-ventilated case. I am focusing on active cooling through the inside end (power plug end) to blow to the case exterior (using the mounting holes on the inside end of the card), as this gets the heat out of the case. Uezi noted above that he is seeing ~47 degrees C. under load with a mounted blower, and that sounds good to me.
Uezi:
Thanks very much for your detailed notes, they will be a big help!
..well letsee, My 0-195 now has three front 120 mm and one side 240 mm intake fans with dust filters There is the standard 120mm exhaust fan in the back and two on the top
Auxiliary fans I am finding for the M40 look like this:
https://www.ebay.com/itm/123949246966
Another vendor is selling just the adapter with no fan for $10 plus $5 shipping. But $7 extra for a good 60mm fan is a pretty good deal.
https://www.ebay.com/itm/GPU-Cooler-Mount-for-Nvidia-Tesla-K80-P-V100-M40-Passive-Cooling-Mount-only/124071246176?ssPageName=STRK:MEBIDX:IT&_trksid=p2060353.m2749.l2649
...yeah that arrangement looks like the best space saving one. another I saw had a hamster wheel fan but it would add even more length to the card and likely not even fit in my massive case. Even with the one above, it would be a tight fit with the front drive bay.. Then again I have 4 intake fans (the large one right by the GPU area) and three exhaust fans.so maybe wouldn't need it.
From the spec drawings and pictures, the heat sink appears to be larger than that of the Titan X. With the shroud removed and good airflow through the case, that may be enough. Removing the shround also gives the option of attaching small fans to the side of the heat sink itself, or postioning a larger fan inside the case to blow directly on the heat sink.
My Tesla and fan adapter are supposed to get here this weekend, and I will try some things.
This seems to be the most practical/economical solution to the VRAM issue. Yeah, I would like to have a TItan RTX, but I would like to have a Neko girlfriend and robot servants too.
I will post pics and findings as I experiment with the card.
...looking forward to it. Again I may have enough airflow in that p-193 to handle it.
Uezi:
Am I correct that you are NOT using a PCIE power connector to power this - you are using the CPU 12V plug?
Thanks!
Are you using the adapter that powers the M40 from two PCIE 8-pin plugs? Thanks!
Thanks for confirming! They are available separately as NVIDIA Graphics Card Power Cable 030-0571-000 for those interested. Some M40 sellers include them, some do not.
Got two M40s in today, waiting on the power adapter cables. Going to try out a couple of more compact cooling solutions to try to fit in my workstation case.
...curious, in looking back at the render times posted by uezi, 5,000 iterations taking 9,834 seconds is about 164 min, or just under 2 hours 45 min. That's fairly long for GPU based rendering. particularly with 3,072 cores. I wonder if the x1 slot on the riser and USB speed has something to do with that as well.
Wondering what size PSU is in the system is that a second one is needed for the Tesla. I have a 750w in the case,
You cannot directly conmpare CUDA across microarchitecture generations. The M40 is a Maxwell card. It has the same chip and CUDA count as the Maxwell Titan X. So you should go through the benchmark threads and see how that compares to newer cards. I'm guessing it comes in somewhere around an RTX 2070.
...my Titan-X is also Maxwell so it is the same generation architecture. I've rarely break an hour and I do some pretty involved scenes.