Delgine 3D Tools & Content DeleD Community Edition
Forums
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

dmf format

 
This forum is locked: you cannot post, reply to, or edit topics.   This topic is locked: you cannot edit posts or make replies.    DeleD Community Edition Forum Index -> DeleD Community Edition
View previous topic :: View next topic  
Author Message
Basiror
DeleD PRO user


Joined: 12 Dec 2007
Posts: 59

PostPosted: Sun Dec 16, 2007 10:16 pm    Post subject: dmf format Reply with quote

I just compared the size of my current scene in .dmf and .dxs format

will you support dmf in the future or do you plan to get rid of it


it would be a pitty, dmfs are way easier to read and much smaller as well
Back to top
View user's profile Send private message
Twixn
DeleD PRO user


Joined: 01 Dec 2005
Posts: 136
Location: Melbourne Australia

PostPosted: Sun Dec 16, 2007 10:34 pm    Post subject: Reply with quote

To be honest I have to agree.

DXS is great for storing all of DeleD's information for DeleD
to use. But for other programs, loading it properly (via proper
xml parsing) takes up huge resources and load times.

Perhaps a binary format is called for? Something more optimized.
Or is that what the plugin system is for Razz

-Twixn-
_________________
Failure is not an option!....It comes bundled with the software.
Back to top
View user's profile Send private message Visit poster's website MSN Messenger
Basiror
DeleD PRO user


Joined: 12 Dec 2007
Posts: 59

PostPosted: Sun Dec 16, 2007 10:48 pm    Post subject: Reply with quote

well you can parse the whole dmf with a few calls to scanf functions in C iteratively
for the xml based formats you need recursive parsers which are a pain and you waste 50% of the memory


regarding a binary format, it has the disadvantage that you have to take care about the endianess if you want to work cross platform, these repeated byte swapping operations may slow down the loader on some platforms with different endian
Back to top
View user's profile Send private message
Twixn
DeleD PRO user


Joined: 01 Dec 2005
Posts: 136
Location: Melbourne Australia

PostPosted: Sun Dec 16, 2007 10:56 pm    Post subject: Reply with quote

Basiror wrote:
well you can parse the whole dmf with a few calls to scanf functions in C iteratively
for the xml based formats you need recursive parsers which are a pain and you waste 50% of the memory


Thats what I meant by proper xml parsing, where the entire model
has to be loaded into memory, parsed into a tree and the model
data extracted from that.Rather than streaming the data from a file,
directly into something usable.

Then again, DXS makes a good intermediate file format. That's
probably what it is designed for.

Basiror wrote:
regarding a binary format, it has the disadvantage that you have to take care about the endianess if you want to work cross platform, these repeated byte swapping operations may slow down the loader on some platforms with different endian


I didn't think of that. However, I'm sure it will still be considerably
less memory and CPU usage than parsing an XML document.

-Twixn-
_________________
Failure is not an option!....It comes bundled with the software.
Back to top
View user's profile Send private message Visit poster's website MSN Messenger
Basiror
DeleD PRO user


Joined: 12 Dec 2007
Posts: 59

PostPosted: Sun Dec 16, 2007 11:16 pm    Post subject: Reply with quote

I personally prefer text based formats for a simple reason,
if something gets screwed you have a human readable format and can correct it in an editor
if your archive header gets corrupted your data is lost.
even id software switched to text base model formats some years ago

that xml boom is over, people prefer easy to parse formats nowadays, many companies go the same way too
Back to top
View user's profile Send private message
Daaark
DeleD PRO user


Joined: 01 Sep 2004
Posts: 2696
Location: Ottawa, Canada

PostPosted: Sun Dec 16, 2007 11:29 pm    Post subject: Reply with quote

Neither format is optimized for quick loading into a game. If you want that, write your own exporter with the plugin SDK
_________________
Back to top
View user's profile Send private message Send e-mail Visit poster's website MSN Messenger
Twixn
DeleD PRO user


Joined: 01 Dec 2005
Posts: 136
Location: Melbourne Australia

PostPosted: Sun Dec 16, 2007 11:34 pm    Post subject: Reply with quote

Basiror wrote:
that xml boom is over, people prefer easy to parse formats nowadays, many companies go the same way too


The boom is probably over, but XML is still widely used. Its easy to
read/edit/create and is very flexible, its just not fast nor easy to
actually parse.

XML is very well suited to very dynamic data (XHTML, certain config
files etc). However, when things are reasonably predictable (such
as a 3D model format), and contain lots of information (such as a
3D model format), and usually very performance orientated (such
as a 3D app), it begins to be less useful.

Daaark wrote:
Neither format is optimized for quick loading into a game. If you want that, write your own exporter with the plugin SDK


Not exactly the easiest thing to do, even with available resources.
And impossible if you want to avoid DXS completely Razz

But still, if no other format works for you its either that or make
a converter to your own ultra-speed format.

-Twixn-
_________________
Failure is not an option!....It comes bundled with the software.
Back to top
View user's profile Send private message Visit poster's website MSN Messenger
Daaark
DeleD PRO user


Joined: 01 Sep 2004
Posts: 2696
Location: Ottawa, Canada

PostPosted: Mon Dec 17, 2007 12:13 am    Post subject: Reply with quote

XML is not hard to parse, nor is it a bad format for DeleD. It allows easy customizations with user defined tags, and the like. It also allows for easy ignoring of new tags, and older versions skipping over unsupported tags, so the file format doesn't have to change and break with every version.
_________________
Back to top
View user's profile Send private message Send e-mail Visit poster's website MSN Messenger
Twixn
DeleD PRO user


Joined: 01 Dec 2005
Posts: 136
Location: Melbourne Australia

PostPosted: Mon Dec 17, 2007 12:27 am    Post subject: Reply with quote

DXS is a good format for DeleD. Its flexible, extensible and
will be backward compatible.

Its a good intermediate format, decent for the plugin system,
stores all of the information DeleD needs in an easy to read/edit
format.

However, its probably not the best for use in performance orientated
3D apps. Playing with the plugin system is good for making your
own/favored file exporter, but one of the simpler alternatives is to make
a converter or use one of the more performance orientated formats such
as .X or even .DMF.

-Twixn-
_________________
Failure is not an option!....It comes bundled with the software.
Back to top
View user's profile Send private message Visit poster's website MSN Messenger
Koobazaur
Member


Joined: 08 Mar 2007
Posts: 22

PostPosted: Mon Dec 17, 2007 4:25 am    Post subject: Reply with quote

Another vote for DMF. I use that to read data into my engine, far easier and faster than DMX.

Daaark wrote:
XML is not hard to parse, nor is it a bad format for DeleD. It allows easy customizations with user defined tags, and the like.


...and it takes >twice as much space due to sheer tag use.
Back to top
View user's profile Send private message
jwatte
DeleD PRO user


Joined: 26 Apr 2006
Posts: 513

PostPosted: Tue Dec 18, 2007 7:18 pm    Post subject: Reply with quote

Honestly, I don't see the problem with the XML. An in-core XML scanner (SAX style) will not use any extra memory, and will run very quickly. One such scanner is XML-SCAN, which I wrote in a few evenings. It's blazingly fast (faster than sscanf() in some cases), uses no extra memory for text storage, and is only as XML compliant as necessary to make sense of a well-formed XML document.

The NuXporter and LevelWalker plug-ins use this scanner, and it works just dandy. If you don't want to write your own, you're free to use the code from NuX/LW; it's all available on GoogleCode.

http://nuxporter.googlecode.com/
http://levelwalker.googlecode.com/

Given that LevelWalker actually builds a full graphical scene of your level, including collision, it should serve as a fine example of how to load the XML data into your game. If you want something better, write your own exporter -- the NuXporter code will likely serve as a good example that you can modify, too.

It's all MIT license, too, so totally free to use without the GPL taint.
Back to top
View user's profile Send private message
Twixn
DeleD PRO user


Joined: 01 Dec 2005
Posts: 136
Location: Melbourne Australia

PostPosted: Tue Dec 18, 2007 10:23 pm    Post subject: Reply with quote

Heh, I won't doubt the greatness of your plugins jwatte Razz they helped
me a great deal with my own plugin (thanks for that Very Happy ).

There is nothing wrong with XML, it does what it was designed to
do very very well. Which is to be a generic, dynamic, non-dedicated
file format which can be adapted for nearly anything.

However, it isn't exactly advantageous for large amounts of
predictable data, like polygons and vertices. For that, a stricter,
more dedicated file format is often better. And DXS requires XML
parsing to do properly, which is still an extra stage.

As for DMF, taking it out or leaving it in doesn't bother me.
I've invested the time in designing my own format and exporter
plugin. However I can see why some people (who may not have the
time to make a converted/exporter) would like to keep DMF
current and usable.

-Twixn-
_________________
Failure is not an option!....It comes bundled with the software.
Back to top
View user's profile Send private message Visit poster's website MSN Messenger
Basiror
DeleD PRO user


Joined: 12 Dec 2007
Posts: 59

PostPosted: Tue Dec 18, 2007 10:38 pm    Post subject: Reply with quote

jwatte wrote:
Honestly, I don't see the problem with the XML. An in-core XML scanner (SAX style) will not use any extra memory, and will run very quickly. One such scanner is XML-SCAN, which I wrote in a few evenings. It's blazingly fast (faster than sscanf() in some cases), uses no extra memory for text storage, and is only as XML compliant as necessary to make sense of a well-formed XML document.

The NuXporter and LevelWalker plug-ins use this scanner, and it works just dandy. If you don't want to write your own, you're free to use the code from NuX/LW; it's all available on GoogleCode.

http://nuxporter.googlecode.com/
http://levelwalker.googlecode.com/

Given that LevelWalker actually builds a full graphical scene of your level, including collision, it should serve as a fine example of how to load the XML data into your game. If you want something better, write your own exporter -- the NuXporter code will likely serve as a good example that you can modify, too.

It's all MIT license, too, so totally free to use without the GPL taint.


you need a xml parser to work with dxs

for dmf you only need sscanf an old c standard library function that is certainly faster than any xml parser around
loading a file is usually done by allocating a huge buffer, load everything in and parse it, imagine files of size of > 64 MB some operation systems don t allows you to allocate more and 64 mb with the new operator at all, you have to mess around with memory mapping in such a case
Back to top
View user's profile Send private message
Paul-Jan
Site Admin


Joined: 08 Aug 2004
Posts: 3066
Location: Lage Zwaluwe

PostPosted: Wed Dec 19, 2007 11:39 am    Post subject: Reply with quote

First of all, we'll stick to supporting .DMF for a while, so don't worry about that too much. It just won't be updated anymore when we add new features. Which is exactly why it might be a good choice at this point.

You have to realize that these are our formats, and our key issue is change. The modeler is a growing application, and whenever we add new features to the modeler, the format changes. Delimited text files like .DMF suffer from the huge drawback that every change to the format breaks 99% of the readers. XML doesn't have the same issues. As DMF is now frozen, it's also not a bad option.

However, we still officially advise everyone to use their own format (either one of the available exported formats, or roll your own). Our formats are optimized for our use (storing everything the modeler needs the store), and they may be very different from the needs of the final application. We acknowledge that support for rolling your own (our plugin system) is currently lacking (documentation, examples, implementation), and will work on that.

Whether your want your final file formats in binary in text is a matter of taste and the requirements of your project. I personally prefer chunk-based binary formats, but that's probably just because I've worked a lot with those.

p.s. One small nitpick: The true problem with endianness of binary files is inconvenience, not performance. Performance wise, byte swapping << text parsing << file i/o (larger files with text-based formats).
Back to top
View user's profile Send private message Visit poster's website
Display posts from previous:   
This forum is locked: you cannot post, reply to, or edit topics.   This topic is locked: you cannot edit posts or make replies.    DeleD Community Edition Forum Index -> DeleD Community Edition All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum