TriplVing's Bullet Vs PhysX Vs ODE performance comparison

Post Reply
TriplVing
Posts: 12
Joined: Wed Jul 29, 2009 4:13 pm

TriplVing's Bullet Vs PhysX Vs ODE performance comparison

Post by TriplVing » Mon Aug 09, 2010 11:23 pm

* ADMIN split topic, originally started here: http://bulletphysics.org/Bullet/phpBB3/ ... f=9&t=5477

One of my other Vs comparisons has been the processing time taken for each time-step of each engine.

i have tried to make each scenario as simple and therefore as similar as possible for each engine, so for this test I just have a sphere falling, with nothing for it to contact, and simply have a bit of timing code that starts before the simulate call for each engine, and ends directly after. I have setup each engine using as many of its default values as possible, and all are set to step for 1/60th of a second. What I am finding is really surprising me, I was under the impression that ODE was considered the slowest of the engines, with Bullet and PhysX being on par, but in reality, I have found that, for 1000 iterations, taken 3 times, all compiled for 64 bit by the same compiler and run on the same machine with the same background processes the average time-steps are as follows:

Bullet: 0.047273ms
ODE: 0.002644ms
PhysX: 0.125971ms

That's really quite significant!

Why is ODE so much faster than both? I assumed as there were no contacts involved at all that primarily the processing overhead would come from the integration method and implementation in each engine, and I was also under the impression that all three now use a second order symplectic scheme of some kind by default? Certainly the error propagation values I have seen from all three would support that.

If anybody more enlightened as to the inner workings of all 3 engines could provide me with a bit of insight, that would be really good.

P.S: I have tried to "turn off" as much processing as I can for each engine, pairing them down to their very lowest form, so I have ensured in PhysX for example that anything like contact point generation and reporting is turned off. I haven't messed around with multi-threading and have just left all engines at their default values, though I don't believe that ODE or Bullet or PhysX multi-thread by default anyway.

Test setup is as follows:

Sphere of density 1 with a starting position of 0,100,0 (x,y,z) and a radius of 10, all engines "initialised" in their default way, code mainly pulled from each respective engines "hello world" example app, though I imagine as there are no collisions, the choice of pruning tree and broadphase etc shouldn't be as important here?

Gravity is set to 0,-9.81,0, I then have a small bit of timing code that uses the windows timing library, totally standard stuff. I then have a while loop that iterates to 1000 in which i have a call to start my timer, then a call to advance the simulation by 1/60, then a call to stop the timer and finally i write the recorded time out to a file stream.

The machine itself is a laptop, Intel T7700 (dual core 2.4Ghz) with 4GB 667Mhz RAM and an 800Mhz FSB.

The libraries have all been compiled for 64 bit, I am using 2.76 for bullet, 0.11.1 for ODE and PhysX system software 9.10.0224 (64 bit).

The test app is of course compiled for 64 bit, using VS 2010. Each was run 3 times, the values were then averaged for each iteration, and then the values I have posted here are the average of the averages.

I mean, I have access to the PhysX dev code (the 50k license) but it doesn't give me access to the integration methods and being honest the general quality of coding seems relatively high, so I'm really quite surprised to see all three engines producing identical positional values, yet there being speed differences in the range of a whole magnitude or two!

Any thoughts would be great.

User avatar
Erwin Coumans
Site Admin
Posts: 4193
Joined: Sun Jun 26, 2005 6:43 pm
Location: California, USA
Contact:

Re: Bullet Vs PhysX Vs ODE Restitution question

Post by Erwin Coumans » Mon Aug 09, 2010 11:40 pm

You are comparing tests will full optimizations turned on, right?

The integration in Bullet is not optimized as it usually doesn't show up in typical profilings, so you might be measuring an un-optimized code path.

Can you provide more detailed profiling data for Bullet, using CProfileManager::dumpAll(); after stepSimulation?
If you want further help it is best to share your test bed for each engine (attach a zipfile to this topic), so we can reproduce your results and investigate further.

Thanks,
Erwin

TriplVing
Posts: 12
Joined: Wed Jul 29, 2009 4:13 pm

Re: Bullet Vs PhysX Vs ODE Restitution question

Post by TriplVing » Mon Aug 09, 2010 11:55 pm

Hi Erwin,

Ok thanks, I don't actually have my test beds at home, they are on my office laptop, so i'll have to get at them tomorrow.

I'll make doubly sure I'm not doing something silly, and then clean up the three frameworks and upload them.

Thanks again.

TriplVing
Posts: 12
Joined: Wed Jul 29, 2009 4:13 pm

Re: Bullet Vs PhysX Vs ODE Restitution question

Post by TriplVing » Tue Aug 10, 2010 7:07 pm

Ok, I made sure that I hadn't compiled Bullet with debug info enabled, and although I was positive i had disabled it, the libs I produced this time round were half the size.

ODE and Bullet are now more comparable, with ODE mostly producing figures in the region of low e-06 and high e-07 and bullet mid to high e-06, certainly closer than before.

I also totally nuked my PhysX SDK and re-installed everything just to make sure I hadn't compiled a dodgy lib or DLL at some point and forgotten about it. Anyway I am now seeing PhysX produce results in the mid to high e-05. So faster than before, but still significantly slower than Bullet or ODE!

I can upload the frameworks if anybody wants me to, but really I suppose my "problem" with Bullet is resolved, the difference between them is now small enough that it can be explained away simply through implementation. As I understand it, ODE is generally a more lightweight library than Bullet or PhysX, but with less features.

I was wondering though, can anybody explain the difference for PhysX? I know its a black box library so anything anyone can offer would be speculation, but I'd be interested to hear peoples thoughts.

Also, let me know if anybody wants me to upload the frameworks.

Thanks again.

User avatar
Erwin Coumans
Site Admin
Posts: 4193
Joined: Sun Jun 26, 2005 6:43 pm
Location: California, USA
Contact:

Re: Bullet Vs PhysX Vs ODE Restitution question

Post by Erwin Coumans » Tue Aug 10, 2010 8:27 pm

Yes, please upload that framework somewhere, or attach it as a zipfile to this forum topic.
As I understand it, ODE is generally a more lightweight library than Bullet or PhysX, but with less features.
The core ODE library isn't necessarily more lightweight: its collision detection and rigid body dynamics code size should be similar to Bullet.

Here are some lacking features of ODE from my perspective:

1) ODE lacks warmstarting of contact constraints, so ODE should have inferior contact/stacking stability than Bullet or PhysX
2) ODE doesn't have proper convex hull collision detection, which is a desired feature
3) ODE, Bullet and PhysX use the same algorithm for the constaint solver (PGS iterative LCP solver) but Bullet's constraint solver is SIMD optimized so it should be faster
4) ODE doesn't have a PlayStation 3 Cell SPU optimized version, Bullet and PhysX do
5) Bullet is written in a more modular fashion than ODE, so it is easier to replace or customize parts of the engine
6) ODE doesn't provide physics authoring plugins for Maya, 3ds Max , Blender etc. Bullet and PhysX do

On the good side for ODE, it has an additional direct constraint solver (dWorldStep based on a Dantzig LCP solver) that Bullet or PhysX doesn't have yet, and its simple C API is attractive to many and makes scripting language bindings easier.

It would be interesting to figure out why your test shows those timings for integration, there might be some issue that can be fixed.

If you want to share more findings, please let us know.
Thanks,
Erwin

Dirk Gregorius
Posts: 874
Joined: Sun Jul 03, 2005 4:06 pm
Location: Kirkland, WA

Re: Bullet Vs PhysX Vs ODE Restitution question

Post by Dirk Gregorius » Tue Aug 10, 2010 9:26 pm

IIRC there is a define in ODE that let's you activate warmstarting. Anyway, the lack of tools and general convex collision detection in the ODE are show stoppers in my opinion.

User avatar
Erwin Coumans
Site Admin
Posts: 4193
Joined: Sun Jun 26, 2005 6:43 pm
Location: California, USA
Contact:

Re: Bullet Vs PhysX Vs ODE Restitution question

Post by Erwin Coumans » Tue Aug 10, 2010 9:41 pm

This warmstarting option in ODE is only for non-contact joints, not for contact constraints. ODE doesn't provide persistent contact point management to store warm starting information, as far as I know.

Thanks,
Erwin

TriplVing
Posts: 12
Joined: Wed Jul 29, 2009 4:13 pm

Bullet Vs PhysX Vs ODE performance comparison

Post by TriplVing » Tue Aug 10, 2010 9:45 pm

Ok, have attached the three frameworks, apologies that the timing code uses a windows header. It was given to me by a colleague and I have used it elsewhere in my work, so I just keep using it to be consistent. I know its not the best way to take critical timings, but it should at least produce comparative results, even if not entirely accurate.

They are all Visual Studio 2008 projects (earliest version of VS I had). In the case of Bullet and ODE, I have included the libraries that I have compiled for x64, for PhysX, it will be necessary to alter system paths from my system to make it compile as well as having the latest SDK (2.8.3 build 21) and the latest system software.

In all cases, only the "x64: Release" profile is properly setup.

I'm very open to the possibility I am doing something quite stupid :) I am a researcher, not software developer and while I like to think I am OK at it, it is not my main trade, rather a tool that lets me do my research.

So I would be very interested to hear any feedback anyone has to offer.

Thanks again.
Attachments
ODEFramework.zip
ODE 64 bit test framework
(885.66 KiB) Downloaded 408 times
PhysXFramework.zip
PhysX 64 bit test framework
(7.61 KiB) Downloaded 374 times
BulletFramework.zip
Bullet 64 bit test framework
(1.8 MiB) Downloaded 437 times

Dirk Gregorius
Posts: 874
Joined: Sun Jul 03, 2005 4:06 pm
Location: Kirkland, WA

Re: Bullet Vs PhysX Vs ODE Restitution question

Post by Dirk Gregorius » Wed Aug 11, 2010 9:07 am

You might be correct since I remember now that the contact constraints are not persistent. I cannot say 100%. Anyway I was refering to this line from the code which gave me the impression that they use warmstarting also for contacts:

//***************************************************************************
// configuration

// for the SOR and CG methods:
// uncomment the following line to use warm starting. this definitely
// help for motor-driven joints. unfortunately it appears to hurt
// with high-friction contacts using the SOR method. use with care

#define WARM_STARTING 1

TriplVing
Posts: 12
Joined: Wed Jul 29, 2009 4:13 pm

Re: Bullet Vs PhysX Vs ODE Restitution question

Post by TriplVing » Wed Aug 11, 2010 5:51 pm

In case anybody is interested to see the graph of my results, I thought i'd attach it to this post.

Image

This was produced using the three test frameworks included 2 posts up, with a fourth added created using a very simple little RK4 integrator I wrote purely to get some baseline figures (it calculates position and rotation only, no collision response etc, but I thought it would be interesting purely as a baseline addition).

500 iterations were performed 3 times for each engine and the results plotted are the average of those 3 values.

ODE really does seem to whip along!

DannyChapman
Posts: 85
Joined: Sun Jan 07, 2007 4:29 pm
Location: Oxford, England
Contact:

Re: Bullet Vs PhysX Vs ODE Restitution question

Post by DannyChapman » Thu Aug 12, 2010 10:34 am

I'm not sure that this is a very fair performance test for the engines! Engines will be optimised for more than one body, and also probably for bodies in contact rather than unconstrained.

For example, when I run the PhysX test case I get timings of around 0.07ms, for one sphere falling.

However, the total time for 100 spheres falling is 0.17ms - i.e. only 0.0017ms per sphere. Obviously that's a huge speedup per sphere, even though there's now lots of collision detection to be done!

Obviously there's an overhead in updating the engine, and whilst that overhead may be much more than the cost of updating a single object in the scene, it would normally be pretty small compared to the overall physics budget in most real simulations.

TriplVing
Posts: 12
Joined: Wed Jul 29, 2009 4:13 pm

Re: Bullet Vs PhysX Vs ODE Restitution question

Post by TriplVing » Thu Aug 12, 2010 12:01 pm

DannyChapman wrote:I'm not sure that this is a very fair performance test for the engines! Engines will be optimised for more than one body, and also probably for bodies in contact rather than unconstrained.

For example, when I run the PhysX test case I get timings of around 0.07ms, for one sphere falling.

However, the total time for 100 spheres falling is 0.17ms - i.e. only 0.0017ms per sphere. Obviously that's a huge speedup per sphere, even though there's now lots of collision detection to be done!

Obviously there's an overhead in updating the engine, and whilst that overhead may be much more than the cost of updating a single object in the scene, it would normally be pretty small compared to the overall physics budget in most real simulations.
Agreed, I understand that. I was just quite interested to see the difference between the engines in such a pared down case. I agree it's more likely that rather than the actual coding standard being better in ODE, it is just structured in such a way that for smaller tests, its better.

What I might do in a minute is run some larger tests as you suggest, maybe 1000 spheres aligned cubically with each row slightly above each other so that everything falls to begin with, and a plane at zero. That might offer an interesting alternative insight and provide a more fair comparison.

I'm not really bashing any engine per se, i'm just trying to do an objective black-box test of them all.

I'll post up the results of said larger tests in a bit.

TriplVing
Posts: 12
Joined: Wed Jul 29, 2009 4:13 pm

Re: Bullet Vs PhysX Vs ODE Restitution question

Post by TriplVing » Thu Aug 12, 2010 4:28 pm

OK,

I have performed some more tests and its quite revealing!

I have re-done the tests from the uploaded frameworks in order to make everything fair and equal. So I have ensured that for all tests I am using a SaP broadphase (set to single rather than multi in PhysX).

The first graph is as before, so this is just a sphere falling due to Gravity, nothing for it to impact. As before, all plotted points are an average of 3 runs on the same machine (T7700, 4GB 667Mhz DDR2 RAM):

Image

The second graph really highlights what DannyChapman says though. In this test there is now a ground plane defined and 625 of the spheres from test 1 are arranged in 5 planes of 5x5, with each "plane" of spheres raised above the other by 2 units, just to ensure that there is movement through the system due to gravity, rather than them all starting in contact. Now we see PhysX and Bullet really steam ahead of ODE:

Image

Quite an interesting set of comparisons for me.

Post Reply