Follow Slashdot blog updates by subscribing to our blog RSS feed

 


Forgot your password?
Close
typodupeerror
×
Technology

FEAD Compressing Compressed Files by 50-75%? 75

Posted by Cliff from the if-it-seems-that-way-it-probably-is dept.
An anonymous reader asks: "I just installed Acrobat Reader and found that it was using FEAD which claims - 'FEAD© Optimizer© significantly reduces the size of application programs on average by 50% (in some cases up to 75%, depending on the specific software), even when they are already compressed with common compression technology like ZIP or CAB.' . It seems that they optimize each application individually at thieir labs. But an average of 50% compression on already compressed binary files seems to be too good to be true. Anyone familiar with how someone may be able to achieve this?"
This discussion has been archived. No new comments can be posted.

FEAD Compressing Compressed Files by 50-75%? More

FEAD Compressing Compressed Files by 50-75%?

Comments Filter:
  • IIRC some old IBM Fortran compiler used to produce very small by using almost nothing but subroutine calls in the actual program producing something kinda like P-code. Maybe they are doing something similar.

    Other than that ? Probably just marketing.

    • Uh, how would this be related? Compressing data files is a lot different from compiling to bytecode, or to a machine language which uses dynamic libraries.
  • This is a common hoax. Maybe 2 years ago, another Slashdot editor posted this hoax. So, it's a repeat hoax for Slashdot, too.

  • It's not compression (Score:5, Informative)

    by photon317 ( 208409 ) on Tuesday June 10, 2003 @07:59PM (#6166492)

    The thing they tout as FEAD is basically a load-over-network-on-demand thingy. They haven't actually developed anything that does compression, they're just storing some of the app on a server somewhere to be downloaded on demand. The hype at their site mislead you, like it was meant to do.

    • Re:It's not compression (Score:4, Informative)

      by bobbozzo ( 622815 ) on Tuesday June 10, 2003 @09:54PM (#6167269)
      When you download Acrobat, it usually will download an "Adobe Download Manager" or something like that.

      That is NOT what is being discussed here.

      Even if you bypass using the download manager, it still uses FEAD to decompress and install AcroRead.

      One could easily disprove your theory by unplugging their net connection during the FEAD decompression... Done... no adverse affect.

      Nonetheless, the installer is VERY slow, and is still bigger than the AcroRead 5.1 installer, which did not use FEAD.

      Making users go through this many steps (download the download manager. run it. wait for it to download. wait for fead...) and slowness is insane.

      • Well, the information on FEAD available by going where the /. submission references shows a pretty picture along with some very convoluted wording that seem to indicate that they're using the word "compression" to describe dynamic loading of peices of the application over the network, thereby "compressing" the original install download.

  • Compression is easy (Score:5, Funny)

    by cybermage ( 112274 ) on Tuesday June 10, 2003 @08:10PM (#6166566) Homepage Journal
    It's decompressing the file that's hard.

    You can compress all your files down to a single bit using this patented two step process:

    1. Discard all zeros.
    2. Use one to represent any length sequence of ones.

    This is as reliable a compression scheme as most backups to tape I've ever seen, and you can fit a huge number of files onto a single floppy.

  • Marketing obfuscation (Score:4, Insightful)

    by jrpascucci ( 550709 ) <{jrpascucci} {at} {yahoo.com}> on Tuesday June 10, 2003 @08:11PM (#6166574)
    I think that it reads as you interpret: if you put some stuff in a .ZIP, it will further compress it. But, on a very close reading, they are only comparing sizes, and not necessarily saying they are compressing the zip file.

    From the article: "Netopsystems specialists combine and customize these tools and processes for each individual software product so that optimal size reduction results are achieved."

    Note the following from the whitepaper: "Usually software producers compress their data by generating cabinet files or the like...Applying a conventional compression tool like WinZip or WinRAR on such data does not lead to appreciable - often negative - results."

    Read strictly, this says what we know: compressing a compressed file generally doesn't work. They aren't saying they compress the compressed file here.

    Note that towards the bottom, they are comparing 'lossless compressed' data to what they do.

    So, here's my bet: they probably do something like crack open a cab or zip, parse a PDF, for example, for 'magic things' that can be ignored without changing the functionality ('lossy' but nothing of significance lost), or take an HTML file and strip all spaces and newlines between tags. Similar things could be done for other file types: Removing quotes and instead, magic-quoting commas in a CDF. Etc, ad inifinitum.

    All in all, it's lame, but so is most software.

    If you have a gigantic amount (hundreds of gigs terabytes) of different files to back up or move around, with so many file formats that you can't keep them straight, then it might be worth it. If you are lazy and it's cheap, it might be worth it. Other than that, I fail to see the real utility here - disk is cheap, bandwidth is getting cheaper, and reasonably assuming the bulk of this data is generated (an adequate assumption), you can do very similar things by fiddling around with the the output formatting in code.

    J


    • > Similar things could be done for other file types: Removing quotes [...]

      When I have files with lots of quotes in them I reduce the size by using single quotes instead of double quotes.

      • When I have files with lots of quotes in them I reduce the size by using single quotes instead of double quotes.
        This is just wasteful as it only applies to quotes. You should use a smaller font so all characters are reduced in size.

    • So, here's my bet: they probably do something like crack open a cab or zip, parse a PDF, for example, for 'magic things' that can be ignored without changing the functionality ('lossy' but nothing of significance lost), or take an HTML file and strip all spaces and newlines between tags. Similar things could be done for other file types: Removing quotes and instead, magic-quoting commas in a CDF. Etc, ad inifinitum.

      Lossy compression eh? Get LZip [sourceforge.net] for all your lossy file compression needs! It can reduce

  • How to do anything (Score:5, Funny)

    by kurosawdust ( 654754 ) on Tuesday June 10, 2003 @08:12PM (#6166578)
    "Anyone familiar with how someone may be able to achieve this?"

    "Lying through one's teeth" comes to mind...

    • I ready the white paper and it looks like they are actually providing a number of products:

      Download on demand:
      Think Quicktime/IE/etc. Download a small download then download the components they want. I expect they'd also use a protocol similar to rsync which makes downloading alot faster.

      Code Inspection:
      This is where they say then can decrease to size of the executable. If you've done progaming then you know that you can make executables alot smaller. Here's some examples: Remove inline macros and make th

  • i know this one (Score:3, Funny)

    by onya ( 125844 ) on Tuesday June 10, 2003 @08:30PM (#6166714)
    by employing the latest in smoke and mirrors technology. they've invented a new mirror that reflects 110% of all light. neat huh?

  • Wow. (Score:3, Funny)

    by Lendrick ( 314723 ) on Tuesday June 10, 2003 @08:33PM (#6166740) Homepage Journal
    That Site (c) is an Eyesore (c). I wonder if these Dipshits (c) realize that all those "(c)" marks make their Site (c) Difficult (c) to Read (c).

  • EXE compressor? (Score:3, Informative)

    by almightyjustin ( 518967 ) <(dopefishjustin) (at) (gmail.com)> on Tuesday June 10, 2003 @08:39PM (#6166777) Homepage
    Sounds to me like an EXE compressor like UPX - they can compress EXE files better than a ZIP archive can (by taking advantage of known aspects of executable files); so by unzipping, EXE-compressing, and re-zipping, one can reduce the size of an already existing ZIP archive.
    • No. If you compress something twice, no matter the methods used, the second time is unlikely to give any benifit.

      • Re:EXE compressor? (Score:3, Insightful)

        by Naikrovek ( 667 )
        wrong. you say "unlikely to give any benefit," but the truth is that it can be beneficial.

        zip an .exe file. or gzip an .so, whatever you want. then zip or gzip (or bzip2 for that matter) the file again - the doubly compressed file will be smaller than the compressed file it contains.

        this is why people zip up movie files on their sites, it does make a difference, and if you only save one meg on your 40 meg movie, and 1,000 people download it, you just saved yourself one gigabyte of transfer fees, during
        • Well, I didn't believe you, so I tried it out. Here's the results:

          -rw-r--r-- 1 tort tort 337648 Jun 11 02:27 libslang.so.1.4.4
          -rw-r--r-- 1 tort tort 149671 Jun 11 02:26 libslang.so.1.4.4.gz
          -rw-r--r-- 1 tort tort 148000 Jun 11 02:30 libslang.so.1.4.4.gz.gz
          -rw-r--r-- 1 tort tort 148051 Jun 11 02:31 libslang.so.1.4.4.gz.gz.gz


          double compressing saved 1671 bytes. On triple compressing, thats when you end up with a net loss. You learn something new ev
          • Another stunt with multiple files.
            First zip, no compression.
            Second zip, zip the first zip.
            Can be significantly smaller than compressing on the first zip.
            • > Another stunt with multiple files.

              > First zip, no compression.
              Or use another uncompressed archive (e.g. tar)...

              > Second zip, zip the first zip. ...and then compress it (e.g. with gzip)

              > Can be significantly smaller than
              > compressing on the first zip.

              Yes. That's why a .tar.gz is usually smaller than a .zip of the same files (modulo speed flags, etc) even though they use essentially the same compression algorithm.

              You do lose something, though; a .zip is indexed so you can easily extract
      • Maybe I didn't phrase that too well. You're correct, zipping the already compressed EXE won't do much. I was using EXE compressors as an example of how it's possible to reduce an existing zip file's size by "50-75%", assuming that zip file contained a program (which the story states). One would probably zip the compressed EXE anyway to include other support files etc. in one file but that wouldn't offer much benefit in the way of compression.
      • Certainly, it's unlikely and unexpected, but it happens far more than you might expect, and a lot depends on the first compressor. Many compressors don't actually compress their own data, for instance ZIP doesn't make much effort at compressing its file table. As a result, if you "zip foo.zip *.gif ; zip bar.zip foo.zip" then "bar.zip" can be significantly smaller than "foo.zip", especially for very large numbers of files. Try looking at the Usenet binaries groups as well - they frequently double compres
        • - There were a lot of errors for scripts etc., but overall disk saving was around 10%,

          Translation : It didn't work anymore, but I recovered 10% of the disk space that it used to use.

          Hell that's easy, just go to any random directory, pick the largest file in there and delete it. Pretty much the same result.
          • Translation: it would appear that you don't understand "strip", like most of the package compilers on the distro release concerned.

            For the non-developers who can't run "man strip"; "strip" removes the *optional* fluff added to a binary executable (including libraries). Since this is is really only useful when debugging the code, something a user doesn't do that often and certainly shouldn't be done on a production box strip will remove it for you. It does not stop the executable from running. When you

            • Naw, I was just joking about the point you made about some of the scripts not working after they had been stripped.
            • Since this is is really only useful when debugging the code, something a user doesn't do that often and certainly shouldn't be done on a production box strip will remove it for you.

              A production box probably does want debug info--when things start going awry there you want to track them down _fast_. Attaching a debugger to the misbehaving app can save tons of downtime.

              Ideally you wouldn't have any bugs on the production machine, but in the real world...

              Sumner
    • On a (very marginally) related note, the same applies to binary patches. When applied to two versions of the same binary, bsdiff [daemonology.net] (which can take advantage of the structure of executable files) routinely produces patches which are 5-10 times smaller than those produced by Xdelta [sourceforge.net] (which can't).

      In short: Executable files are far more than just streams of bytes.

  • Compressing data exe compressors don't (Score:4, Interesting)

    by TheSHAD0W ( 258774 ) on Tuesday June 10, 2003 @09:19PM (#6167055) Homepage
    When you use an executable compressor, like PKLITE, on an executable file, it can't compress all the data. This is because EXEs will dynamically load more data, and if that data is compressed, the code can't read it.

    I suspect these guys are going in and manually altering the code to perform a decompression. This would certainly produce a benefit.

    Here's something for you to try: Take an executable and zip it. If it compresses, then there's probably SOME give in it. And most executables I see are compressable.
    • The operating system relocates the EXE when it's loaded. EXE's don't (normally) load data from themselves, they load them from external overlays. As long as the execution point points to where the PKLITE code is, it will run fine.

      --Dan

  • Hmm... (Score:3, Interesting)

    by bobbozzo ( 622815 ) on Tuesday June 10, 2003 @10:03PM (#6167330)
    I just RAR'd (RAR 3.11 with -m5) my AcroRead folder... it came out 300k bigger than the 16MB full installer...

    Using ZIP -9 gives a 20MB file.

    So, FEAD offers slightly better compression. (I know there's other crap, including the installer, registry settings, icons, ...)

    Still, is it worth the annoyance of the greatly increased install time?

    Also, how is FEAD saying they are 50% better than other compressors?

  • I have no idea how FEAD works, but here's how I'd do something similar:

    A large portion of shipped executables are blocks of standard code from the compiler. If you're using a Microsoft compiler, you can strip out the standard chunks and pull those chunks in from the binaries that are already in Windows.

    If you're using another compiler, you can still probably do the same kind of thing: some intelligent block compression with the included code that'll do better than the "dumb" compression from "zip" or oth
    • 'combined with an "expander" app'

      Otherwise known as JIT...
    • A large portion of shipped executables are blocks of standard code from the compiler. If you're using a Microsoft compiler, you can strip out the standard chunks and pull those chunks in from the binaries that are already in Windows. If you're using another compiler, you can still probably do the same kind of thing: some intelligent block compression with the included code that'll do better than the "dumb" compression from "zip" or other algorithms.

      Actually, this is what the "sliding dictionary" is in LZW

      • A lot of compilers do tricks to make code faster. Not many do things to explicitly make code smaller.

        Actually, it's common to have compiler switches to optimize for either space or performance, and even feature-specific switches

        And a lot of the time smaller _is_ faster--getting more into icache saves costly trips to RAM. This is becoming more and more true with modern processors, though there are still obvious space/time tradeoffs in many cases.

        Sumner

  • > It seems that they optimize each application individually at thieir labs. But an average of 50% compression on already compressed binary files seems to be too good to be true. Anyone familiar with how someone may be able to achieve this?

    Maybe they're just removing the bloat. I've read on comp.risks about a guy who disassembled Windows regedit and found embedded strings and even images, which were not actually used in the application program.

    But the link is not at all clear about what they are actua

  • Statistical encoders (Score:2, Insightful)

    by afroborg ( 677708 )
    I couldn't say for sure, but it's possible that theyre just using a better coding scheme. ZIP et al use (as far as I know) variations on the LZ type compression algorithms. These are fast, but definitely not the best entropy removal methods available. Arithmetic coding OTOH is very effective, removes more entropy than LZ, LZW, or Huffman, but is slow because it needs to collect statistics on the entire file before compression. I dunno about decompression speed though Arithmetic coding is patented though
    • entropy removal

      ??? - compression introduces entropy . . . a well-ordered file (little entropy) can be heavily compressed. The compressed version has more entropy per n bits than the original file.

    • I agree. Although the claims attributed to FEAD still sound much too good to be true on average, data compression has improved in the past decade or so with techniques like Prediction by Partial Match with unbuonded length [computer.org], made more practical by Esko Ukkonen's [helsinki.fi] algorithm (published in the early 90's) for constructing suffix trees [ttp] in linear time and linear space, making it much easier to find repeating substrings, and the Burrows-Wheeler transformation (discovered in the 80's, published in the early 90'
      • I will, however, provide this link to bwtzip an experimental compressor covered by the GNU General Public License that uses the Burrows-Wheeler transformation

        bzip2 (included in most Linux distributions and well past the experimental phase) also uses a Burrows-Wheeler transform (with Huffman coding).

        bzip (not 2) used BW with arithmetic coding but was withdrawn because of potential patent problems with that combination (the bzip ari coding wasn't LZW and no concrete patent on bzip's ari coding was known, b
  • AFAIK upx does pretty much the same thing, for free

    http://upx.sourceforge.net

    it generally gets 50-75% too. IIRC it make a really fast (faster than a HD read) decompressor prepended to a compressed program.

    The only thing I can think of to do better is to actually rearrange the binary in a more efficient order / go in at the assembler level and replace any repeated 5 instruction or more sequence with a function call.

Slashdot Top Deals

Two can Live as Cheaply as One for Half as Long. -- Howard Kandel

Close