some kind of compressed images support

[xrmb]

I know DT is there to emulate cd-images, but one thing I hate is that I have to have them uncompressed, since no supported format supports compression.

At the moment I have like 34-cd images of a clipart-collection on my hdd, pretty good waste of space, because every image can be compressed from 650mb to <100mb with gzip!

I can't use ntfs-file compression because its a software raid.

Ok, here is my idea

If I'm right every cd-image consists of many blocks (2048, 2352 or what ever bytes in size);

since i don't wanna loose the amazing speed of DT I would compress every sector by its own, gzip and bz2 are pretty good in compressing small amounts of data. So all you would need is a second file for faster seeking in the compressed image, since the compression will change position of the sector in the image file. A converter from ISO to this gziso format should be easy to write...

If somebody is interessted in a comparison between rar/zip-packed, ntfs-compressed isos and this idea I could write a small program that estimates the compression (even if i dont know everything about iso, but enough for the quick-shoots).

And one more point... gzip and bz2 are fast and free. Only disadvantage, we would have to specify our own format, and I don't know if that is in the scope of the DT developers.

Maybe, in a near future (tomorrow) DT will have an open plug-in system that allows us to write our own cd reader

thanks for reading, ever comment is welcome.

[DevX]

Maybe. If you want to contribute to development of such compression - mail me and we can discuss it.

[xrmb]

the first results

ok, took me a little bit more time than expected, but you get details and tools...

I don't know where to start...

Ok, I grabbed some cds and a dvd from my desktop and created isos, 2048 bytes sectors.

1. games: civilization 3, and freelancer (btw I own these games!)
2. data: a clipart cd (with wmf files)
3. applications: office xp and the vs.net 2003 60day-trial-dvd

Then I took some common used compressors (rar and zip) and packed all images as good as possible to get the compressed size.

After that I picked two free available compression algorithms we could use for this project:
gzip = pretty old, but still good, excelent for small data-blocks
bz2 = almost like rar in compression, but much faster

And I compressed the images with these tools too.

Now I there are some points I was thinking about. I assume the DT needs the data somehow blockwise (like it is in the image or on cd);

that means one big stream like rar, zip aso create are useless. Every block in the compressed image has to be compressed by its own. Only this way you can seek to every block w/o reading and expanding data (useless data).
Since the compression is not that good in single blockmode, I started playing with compressing 16 blocks (32... 64... aso). From 1 to 16 is big gain in saved bytes, and there could also be a speed advantage, since in most cases more than one block is requested. In every case you wanna access one single block you would have to expand 16, 32... block, could be a waste of resource, we have to find the best value for this.

Take a look at the linke results files, or play around with the gziso (analysis tool) to get an impression.

Here are some things i figured out so far:
- the difference between gz and bz2 becomes bigger with higher number of merged blocks
- single block compression is really worse, but at 16-block compression you are already close to zip (with gz) and rar (bz2)
- speed is no reason, maybe for creating a compressed image, but decompression is in both case lightning fast

things to think about:
- now a block/merged blocks can be either compressed or uncompressed, maybe you should be able to set a threshold, like if the compression only brings 5%, store it uncompressed
- for the file format I thinking about something like png, jpg aso. like: <tag><size><crc><...data...>, but only for the header, not for the image data, has a big advantage for enhancing the format and being backward compatible, trust me... I know what I talking about


ok, enough for now... I'll answer every question, and I thank for every hint...

links:

http://home.attbi.com/~xrmb/gziso/compare1/compare1.html the results

http://home.attbi.com/~xrmb/gziso/bzip2.zip bzip2 compressor (win32 exe)

http://home.attbi.com/~xrmb/gziso/minigzip.zip gzip compressor (win32 exe)

http://home.attbi.com/~xrmb/gziso/gziso-v0.1.1-bin.zip gziso, the analyzer tool I wrote for details (win32 exe)

http://home.attbi.com/~xrmb/gziso/gziso-v0.1.1-src.zip the source code for courios people

http://home.attbi.com/~xrmb/gziso/gziso-results.zip details gziso result file from my test

http://home.attbi.com/~xrmb/gziso/compare1.sxc numbers and charts in a StarOffice spreadsheet

http://home.attbi.com/~xrmb/gziso/compare1.xls numbers and charts in an Excel spreadsheet


upcoming things:
- compare with ntfs compression
- make source code ansi-c
- enhance gziso to write the images
- add an audio cd to comparison


I hope you guys like the work I've done, ok... I'm a little bit upset about the week compression of all the available algorithms But thats out of my range...

[DevX]

Yes, you've done great work, thanks!. And it is clearly seen that there is almost no gain in compression of typical games CD - they are usually already compressed. That is main point why compression of images we never on our priority list. It would be nice if you could make som NTFS comparisons! I guess we need to pin this topic so all interested could see it.

[xrmb]

the day after... finally I know what I've the P4 3.0ghz for 100% load for 18h...

here what I added:
- ntfs compression stats
- more isos
- more stats


ntfs compression stats:
well, its a freebe from microsoft, worst compression possible


more isos:
I added Diablo 1, because this game only copies 2mb from the cd to the hdd... well, but Blizzard knows how to compress data

Added a MSDN (documentation) version too, thats also something that you can use from cd, not much to get out of it :'(

And the Autoroute Data-cd, another cd you dont need to copy on the hdd... Microsoft can compress data if they want...


conclusions:
The gz-compression with 16 blocks merged seems to be the best and fastest solution, not as good as rare, but closer than ntfs compression. On my sample iso you could save 22% with this, its almost 2gb out of 8.5gb. Another highlight (I really like);

is, that the biggest benefit is on application-install cds, like office or visual studio, because windows always wants them to reinstall something.
There was no surprise that the clipart cds are melting like ice in the sun...
For the games I didnt expect a different result, it seems anyway so that every game has to be copied for the cds entirely


@Venom386: well this idea wasnt about saving some bytes on game-cds, I have the originals, so I only need them to install... but its different with data and application cds...

ok, as always, read and ask... btw, a ungziso tool to rebuild the iso to burn it will be provided if needed It anyway only looks like a few lines of code...


new links:
http://home.attbi.com/~xrmb/gziso/compare2/compare2.html the updated stats page

http://home.attbi.com/~xrmb/gziso/compare2.sxc stats for star office

http://home.attbi.com/~xrmb/gziso/compare2.xls stats for excel lover

[PixelaseR]

Just outta curiosity, is this GZISO idea even been considered or worked on yet?

[DevX]

Support for compressed files is planned later as special file access plugin system (parallel with image access plugins).

[PixelaseR]

Support for compressed files is planned later as special file access plugin system (parallel with image access plugins).

I see....and this, I am guessing, won't be done anytime soon?

[xrmb]

I see....and this, I am guessing, won't be done anytime soon?

once they released a version with plugins they can get my full support for compressed iso format, I'd like to write the converter-tools or something like this

[PixelaseR]

I see....and this, I am guessing, won't be done anytime soon?

once they released a version with plugins they can get my full support for compressed iso format, I'd like to write the converter-tools or something like this

I'd love to help with this plugins, though I guess I am gonna have to do some research on ISO formats...lol

Of course....unless they can implement a global function that will return raw ISO data which can then be converted by the plugin. For example, ISOData readISO(FILE* file)...hehe

[DevX]

File access plugins will work independetnly from image plugins - they will not know what is ISO, CUE etc. They will handle only pure access to blocks inside file. Please also not that most likely all these compression plugins will be written as system drives (*.sys) - no Win32 API can be used. During image access no switch from Ring0 to Ring3 will be allowed.

[PixelaseR]

File access plugins will work independetnly from image plugins - they will not know what is ISO, CUE etc. They will handle only pure access to blocks inside file. Please also not that most likely all these compression plugins will be written as system drives (*.sys) - no Win32 API can be used. During image access no switch from Ring0 to Ring3 will be allowed.

Actually, on 2nd thought, I shouldn't be talking about conversion with D-Tools...lol...my bad

I was actually talking about the converter...bleah....two different programs.

I'd like to help with the plugins...but it looks like I am gonna be lost with the fact that I have no idea what Ring0 & 3 is...lol....off I go to do some research...

[xrmb]

The protected mode offers 4 different 'privilege
levels' (ranging from 0..3, aka ring0..ring3). Userland applications
are usually executed in ring3. The kernel on the other hand is executed
in the most privileged mode, ring0.

damn, I learned all that in schools a view years ago, but never used it again... stupid c++, but at least some guys here still know how the system works, no its not the start menu I'm talkin' about

[DevX]

I will try to make design as simple as possible to reduce number of kernel code. All this will need some work and testing. The main limitation is 9x OS - on 2000/XP it is practicaly possible to avoid kernel code at all (at some performance hit maybe, but i don't think it matters at all as anyway performance will be lost during decompression).

[PixelaseR]

The protected mode offers 4 different 'privilege
levels' (ranging from 0..3, aka ring0..ring3). Userland applications
are usually executed in ring3. The kernel on the other hand is executed
in the most privileged mode, ring0.

damn, I learned all that in schools a view years ago, but never used it again... stupid c++, but at least some guys here still know how the system works, no its not the start menu I'm talkin' about

Actually....now that you mention it...I do remember having learnt about privileges when I took Windows Programming. I just didn't know they were called "ring". :P Actually, in reality, I haven't touched C/C++ for a long time since for some reason, everyone went for Java all of a sudden. I just started picking C/C++ along with C# back up about 2 months ago....sigh....industry transformation can be devastating....

Now....as for not switching from ring3 to ring0, you mean not using any Windows kernel code but completely "rewrite" the entire ASM code to emulate communication of DVD drives as well as decompression? I believe that would be somewhat very complicated and may takes a long time to optimize wouldn't it?