some kind of compressed images support

[PixelaseR]

I will try to make design as simple as possible to reduce number of kernel code. All this will need some work and testing. The main limitation is 9x OS - on 2000/XP it is practicaly possible to avoid kernel code at all (at some performance hit maybe, but i don't think it matters at all as anyway performance will be lost during decompression).

Regarding performance hit.....maybe after the code is done, minor code optimization can be done to pull it back up a little bit. I certainly don't think the performance hit would be that great.

[DevX]

It can be done in C (not ASM) but it is complex task indeed - experience in writing drivers needed.

[CeeJay]

Xrmb : Could you do test with my favorite compressors too ?
7-zip - http://www.7-zip.org/ - For superb compression
LZOP - http://www.lzop.org/ - For superb speed
(The other datacompression libraries by Markus Oberhumer might also be worth a look)

Both are opensource.

Also note that while the g-zip rutine in 7-zip is about 10% more efficient that G-zip itself the really interesting format is it's own 7z format.
The only problem I can see with it is that it's hard finding the perfect settings for your specific file with so many options available.

0=BCJ2 1=LZMA 2=LZMA 3=LZMA b0:1 b0s1:2 b0s2:3 1d=25 2d=20 3d=20 1mf=bt4b 2mf=bt2 3mf=bt2 seems to work really good

Also you are only compression ISO images, try also to compress CloneCD and Alcohol 120% images , because while it's true that gamecontent can't always be compressed that well , copy-protection data like custom subsectors and etc can be compressed REALLY well.

I did some tests on a CloneCD image a few years back when I first suggested that Daemon should support compressed images and the .sub was compressed down to something like 8% of the original size

BTW they told me I was nuts and that there would be no gain from it since games were compressed already and it would never happen and if they were ever going to support compressed images they would make their own compression algorithm that would be so much better than anything else seen before.

Ofcourse they said something along the same lines when I suggested the analog audio feature and look ! It's in daemon now *g*

I suspect that audio CD's can be compressed to something like 50% of the original size ..
FLAC is a very good codec for that (since it's lossless and backed by Xiph.org)

http://flac.sourceforge.net/

It is ofcourse also opensource.

And while we are on the subject of custom format plugins consider a mp3/ogg vorbis -> audioCD plugin.
I also suggested this years ago - It's intent was to make it possible to hear the music YOU wanted in games that otherwise didn't support custom music and loaded its own music from an audio CD.
I think the developers response back then was that it was completely unnecessary because you could just burn the files to a CD and use it instead *g*

[Jaime]

The amount of compression always depends on the data entrophy level. Entrophy can be assimilated to the concept of disorder (in the sense of lack of unicity);
so compression reduces the entrophy level, reduces the disorder.

Audio data is not as disordered as you may think, in fact, if you take a mono wave file containing music or voice (without silence);
the compression level that can be achieved will be very low. Losless audio compressors capitalize mainly on the concept of 'joint stereo', i.e. in the differences between the channels, so the results will depend on the kind of music/audio data that is processed.

Typical results can vary betwen 45-80% of the original size, more towards the higher values for pop/rock music. If you want to achieve good ratios there's no other way than using lossy compression formats (mp3/mp4/aac/...).

But in my oppinion this is irrelevant, most people will desist on using compression when they see that compressing their game CDs almost gets them no savings at all. The interesting thing is that implementing a flexible plug-in architecture for data processing will open a new range of possibilities, that are good for but not limited to compression; data encryption is a very interesting field that can open new markets for this software.

Following with the previous reasonement, one of Daemon Tools' strong points is the ability to filter data coming from CD/DVD drives, so why not make this feature expandable so data decryption (or other kind of things) can be performed in real-time from physical drives?.

I would suggest to take a look at the BestCrypt's development kit that can be found at http://www.jetico.com, BestCrypt has a kernel mode API and plug-in system.

[LocutusofBorg]

a very good idea, after all new plugin method will provide us with many new concepts and advantages, that I'm sure about

[ivan]

paragon's cdi

Maybe you should add support for paragon's compressed image format (cdi). Paragon CDI is the native cd image format used by for the Paragon CD-ROM Emulator program to create backup copies of cds. The special thing about Paragon CDI is that it supports data compression.

I'm not competent in cd image file formats, but I think that Paragon CDI is different from DiskJuggler's CDI, because when I try to load a Paragon CDI in DT the only thing I get is an error message.

Paragon homepage: http://www.cdrom-emulator.com/

I hope this helps you.

[Benji99]

I also eagerly wait the compression iso feature, but I have a slightly different advantage in mind.

Basically I plan in the near future to rip every single cd I own to the harddrives (those are anything from application cds, to games, training cds etc...).

Basically I have a few problems (albeit very minor problems) with this, one of which was already mentioned, I would be wasting space, and If I rar them, I have to decompress them before I use them, which would suck.

THe other issues I have is that since I like to use the ripping program which I think is best for the job, clone, BW, cdrwin, DJ, etc... I will use several programs, and this will result in various different formats which will look unclean, but at the same time, clone creates 4 files, cdrwin 2, etc..., It would be nice to have a universal "container" compressed or compressed where 4 files would look as 1 on the file system, and they will use the same extension regardless of the actual image format used within.

I know my problems are very minor, but I'm anal retentive, and would love to see compression added to daemon tools, which would solve practically all my problems, hell I'd even be able to burn the cds directly without decompressing first...

Benji99

[shawn8888]

I am wondering if it is difficult to apply this feature. (Support Zip or Rar)
Could it be possible in the next version?

[The_Morph]

One of the biggest problems will be that several of the mentioned compression algos are not documented nor available for free. I stumbled across an almost perfect replacement for rar recently: sqx. sqx is a freely available compression/decompression library from Rainer Nausedat (www.sqx-archiver.org) which comes will full source code and a complete SDK. This will surely ease up things a lot. Even more, SQX comes with support for archive encryption, so no need to create another plugin. (And just for the records, I'm not connected to SQX in any way, I just like it almost as much as my beloved winrar )

If I remember correctly, the daemon drives have 512 kbytes cache by default so why not compressing chunks of 214 blocks. Data is read sequentially in most cases anyway so there wouldn't be a large overhead but better compression in return. Why 214 blocks? A cd sector can have 2448 bytes maximum size, depending on the image format ofcourse.
A raw sector has 2352 bytes, plus 96 bytes subchannel data makes 2448 bytes. As we have to take care about the maximum allowed cache size, we will have to stick to 214 blocks per chunk or use variable chunksizes depending on the current image data.

Audio CD support will be a bit more difficult, at least for lossy compressors like mp3. Anyone who reads the lame discussion boards will know why.

I haven't had a look at FLAC and other lossless compressors, so I don't know how accurate chunk compression will be with these compressors, however, audio compression may turn out to be a real beast.

By the way, great idea with the image plugin system. Is any documentation about it available? I don't have much time these days

[zagnut]

this whole compressed iso concept is interesting... i've considered it in several respects, particularly with regard to:

1) transparency to the user
2) responsibility of this product's developers and this particular product
3) demand
4) comparable solutions/alternative approaches

with respect to .1, existing formats (rar, gzip, bz2);
none of them have really been designed with random access in mind, so the compression is always quite block-oriented (ie, files are compressed as whole chunks for a size that is often chosen for how it affects compresability -- ex, by file, by set of files, or by some maximum size premited by memory)... while this helps get a good ratio, it doesn't make for quick/easy decompression (random access and speed)... so, just like with Zipfolders, i fear existing formats will tend to require long waits before access... this is less of an issue on pimped-out boxes, but not everyone has 2gb ram and 4ghz idling... interestingly, flac HAS been designed with decompression speed/random access in mind AND it gets a better ratio than winrar on a 700mb audio cd image i tried the other day... i wonder... has anyone tried compressing data with flac? i might have to...

with respect to .2, i think it's hard to justify the suggestion that dtools be responsible for compression (or at least the definition of a compressed format -- tho if another app/group did, it might be worth simple SUPPORTING it... but no such format exists, and surely competition would quickly negate the effort spent as the first...)... also, if anything was definied it would have to be opensource in nature since surely dtools can't be turned into a conversion app -- that's something for another program to do... dtools just deals with images... any broaders scope would turn this nice small util into a 4bm fatso... the open nature of such a scep would lead to competing implimentations and less advantage.

with respect to .3 the demand is there, and growing... ppl are getting sick of discs, and want a good way to virtualize them. this - lack of compression - is becoming representative of a glass ceiling for many disc junkies; i mean: even 100gb drives fill up quickly when files are 600+mb each...

with respect to .4, as was noted there ARE file-based solution like ntfs, and just because it doesn't work for one person with software raid, doesn't mean it's not a relatively effective solution for others... i mean, even zipfolders would work if you could cope with the lag (in fact, who's to say the under-performing ntfs compression wasn't the lesser of evils... it could be high-compression and lots of lag... i suspect ms considered that issue thoroughly...). so there are options... mp3 is an option, and we've already been told mp3+cue is not an option... i think that's an appropriate choce... if dtools has to support every encoding, why not uuencode? that get's rediculous... rediculous to develop OR support.


all in all i think it's a little too lofty a goal for a utility... this is very generic stuff we're talking about... stuff that shouldn't even come into play in a utility that is really just simplifying the "now" of comuting (discs, copy protections, etc...)... why should it be trying to apply an "architecture"... it seems like the wrong place... an os with compressed files alla ntfs is the place for that... anywhere there is a platform there is a place for that... my question to the developers is this: is dtools a platform?

my opinion is that dtools is presicely where it ought to be... making virtual cd emulation fun... i think the benefit of it comes from the specific nature of it... support only for images to mount...

that said, i feel justified in suggesting in a previous post that dtools might consider supporting flac as an "image to mount"... it may not serve the data side, but it has the [growing] potention to serve to audio side....

[fantc]

Has anyone yet considered the linux compressed iso file image system that KNOPPIX uses?

http://www.knoppix.net/docs/index.php/cloop

http://developer.linuxtag.net/knoppix/sources/cloop_2.01-1.tar.gz

The file format consists of a header, an index table for block translation and of n blocks compressed with zlib at a given original block size.
That seems very simple, and it's easy to create compressed iso's from non compressed iso's.

Where can I get information, how these DLLs in the [image] directory of the daemon tools work? What interfaces must they provide?
I do not have many experiences in c/c++ programming (I mostly deal with java). Any help is appreciated.

[DevX]

Mentioned DLLs currently handle only images without comression. their task is handle information specific to individual formats.
Compression handling is planned to do in other modules (Plugins\Files)
This design is under development.

[fantc]

Hello,

thanks for the information. That's what I thought, but I would like to know more about the interfaces of those DLLs, to try a bit with other proprietary iso images. I also would like to know more about how to write a plugin. Can you provide such information?

[fantc]

I mean a plugin that can be placed in Plugins\Files, or is that not possible at the moment?

[LocutusofBorg]

it is possible. You should write an email to me
locutus@daemon-tools.cc

(only experienced developers please)