Jump to content
Excelsior Forums
Sign in to follow this  
Linuxhippy

Please make JET compatible with upx

Recommended Posts

Hello,

Thanks to the great work which happened in the past the runtime surrounding the executable is now very small, for a small swing-application I created I came to:

* 12,1mb for the executable

* about 6-8mb for the other stuff

Especially when deploying the app over network this would generate a lot traffic which would slow down the application speed. I had some very impressive results with UPX http://upx.sourceforge.net/, an on-the-fly executable (de)-compressor which is able to compress the 12,1mb executable to a handy 4,1mb file.

Sadly the executable dies with an Segementation fault. I also had some negative tries with other packers and older versions of JET.

It would be great if JET-5.1 could support executable packers.

lg Clemens

Share this post


Link to post
Share on other sites

Excelsior JET generates executables that are perfectly valid from the operating system standpoint, otherwise they would not load. So it is the executable packers that have to be fixed.

There is a chance that we'll implement our own packer, but I cannot commit to any particular availability date.

Share this post


Link to post
Share on other sites

I really believe this is a problem of the packers, however wouldn't it be great if the reason why all five packers I tried did not work and give Excelsior-Customers the possibility to pack their global-optimized apps to about 30% of their original size? They would even load fast because of less disk-IO :-)

Well as far as I know a In-Jet packer will again only provided to enterprise versions which means $$$ for avaerage developers :-/

The original applications runs well under wine, so I tried the compressed one there too and it segfaults like on windows.

Maybe the debug-output can be helpful:

[ce@localhost jode]$ cd Jode
[ce@localhost Jode]$ wine Jode_compressed.exe
libGL warning: 3D driver claims to not support visual 0x5b
wine: Unhandled page fault on read access to 0x00000008 at address 0x71d279 (thread 0009), starting debugger...
Unhandled exception: page fault on read access to 0x00000008 in 32-bit code (0x0071d279).
Register dump:
CS:0073 SS:007b DS:007b ES:007b FS:0033 GS:003b
EIP:0071d279 ESP:0033fea0 EBP:0033fea0 EFLAGS:00010203(   - 00      - RI1C)
EAX:00b82cb8 EBX:7eec3884 ECX:00000000 EDX:004001c9
ESI:00b82cb8 EDI:00b82cb8
Stack dump:
0x0033fea0:  0033fed0 0071e320 00b82cb8 0033febc
0x0033feb0:  0033fec0 0033fec4 0033fec8 00b82cf8
0x0033fec0:  00b82cfc 0033feec 009b639b 00400000
0x0033fed0:  0033feec 009b63a1 00b82cb8 00000000
0x0033fee0:  0033fef8 7ffdf000 00401000 0033ffe8
0x0033fef0:  007ad2c5 00400000 00b82cb8 007ad270
Backtrace:
=>1 0x0071d279 in jode_compressed (+0x31d279) (0x0033fea0)
  2 0x0071e320 in jode_compressed (+0x31e320) (0x0033fed0)
  3 0x009b63a1 in jode_compressed (+0x5b63a1) (0x0033feec)
  4 0x007ad2c5 in jode_compressed (+0x3ad2c5) (0x0033ffe8)
  5 0xb7e5f547 (0x00000000)
0x0071d279: movl        0x8(%ecx),%edx
Modules:
Module  Address                 Debug info      Name (53 modules)
ELF       1a1000-  1e7000       Deferred        libsepol.so.1
PE        400000- 115e000       Export          jode_compressed
PE        400000- 115e000       Export          jode_compressed
PE        400000- 115e000       Export          jode_compressed
PE        400000- 115e000       Export          jode_compressed
PE        400000- 115e000       Export          jode_compressed
PE        400000- 115e000       Export          jode_compressed
PE        400000- 115e000       Export          jode_compressed
PE        400000- 115e000       Export          jode_compressed
PE        400000- 115e000       Export          jode_compressed
PE        400000- 115e000       Export          jode_compressed
PE        400000- 115e000       Export          jode_compressed
PE        400000- 115e000       Export          jode_compressed
PE        400000- 115e000       Export          jode_compressed
ELF     4a1a2000-4a2a4000       Deferred        libx11.so.6
ELF     4a2a6000-4a2b6000       Deferred        libxext.so.6
ELF     4a2b8000-4a2bb000       Deferred        libxinerama.so.1
ELF     4a2bd000-4a2c6000       Deferred        libxrender.so.1
ELF     4a2c8000-4a2cd000       Deferred        libxfixes.so.3
ELF     4a2cf000-4a2d9000       Deferred        libxcursor.so.1
ELF     4a2db000-4a2df000       Deferred        libxrandr.so.2
ELF     4ae7d000-4af8f000       Deferred        libwine.so.1
ELF     4b732000-4b737000       Deferred        libxxf86vm.so.1
ELF     4b745000-4b7b3000       Deferred        libgl.so.1
ELF     4d4d1000-4d500000       Deferred        libfontconfig.so.1
ELF     4d76a000-4d78b000       Deferred        libexpat.so.0
ELF     7bf00000-7bf03000       Deferred        <wine-loader>
ELF     7c94e000-7c963000       Deferred        midimap<elf>
  \-PE  7c950000-7c963000       \               midimap
ELF     7c963000-7c98b000       Deferred        msacm32<elf>
  \-PE  7c970000-7c98b000       \               msacm32
ELF     7c98b000-7c9a3000       Deferred        msacm32<elf>
  \-PE  7c990000-7c9a3000       \               msacm32
ELF     7c9a3000-7c9e0000       Deferred        wineoss<elf>
  \-PE  7c9b0000-7c9e0000       \               wineoss
ELF     7c9e0000-7ca73000       Deferred        winmm<elf>
  \-PE  7c9f0000-7ca73000       \               winmm
ELF     7ca82000-7ca9f000       Deferred        imm32<elf>
  \-PE  7ca90000-7ca9f000       \               imm32
ELF     7e412000-7e672000       Deferred        i915_dri.so
ELF     7e83e000-7e8d1000       Deferred        winex11<elf>
  \-PE  7e850000-7e8d1000       \               winex11
ELF     7e9af000-7e9f8000       Deferred        advapi32<elf>
  \-PE  7e9c0000-7e9f8000       \               advapi32
ELF     7e9f8000-7ea97000       Deferred        gdi32<elf>
  \-PE  7ea10000-7ea97000       \               gdi32
ELF     7ea97000-7ebe3000       Deferred        user32<elf>
  \-PE  7eab0000-7ebe3000       \               user32
ELF     7ee09000-7ef3c000       Deferred        kernel32<elf>
  \-PE  7ee20000-7ef3c000       \               kernel32
ELF     7ef63000-7f000000       Deferred        ntdll<elf>
  \-PE  7ef80000-7f000000       \               ntdll
ELF     b7cf3000-b7cfe000       Deferred        libnss_files.so.2
Threads:
process  tid      prio (all id:s are in hex)
00000008 (D) Z:\home\ce\jode\Jode\Jode_compressed.exe
        00000009    0 <==

Share this post


Link to post
Share on other sites

I also opened a thread at bitsum.com (the autors of PECompact) and uploaded some test executable created with jet5-beta2 to:

http://www.bitsum.com/smf/index.php?topic=457.0

http://stud4.tuwien.ac.at/~e0625701/excelsior_test.zip

I also openend a bug-report at sourceforge for upx: http://sourceforge.net/tracker/index.php?func=detail&aid=1721749&group_id=2331&atid=102331

Maybe some exchange could take place, I am sure the packer-authors would like to fix the problem, but I guess its quite hard for the to figure out why executable generated with excelsior-jet fail. Maybe an excelsior-engineer could spend 10min figuring out whats going an and describes the problem?

By the way I was able to compress all dlls/exes in the rt-directory and with some (illegal) hand-tweaking (removing color-management profiles and some other stuff) I was able to reduce my "rt"-directory to 3,94mb :-)

lg Clemens

Share this post


Link to post
Share on other sites

Compressing an executable file has a big disadvantage: it will not be mapped to memory by the operating system and therefore your application will need more RAM. Moreover, if you run two copies of the application, or two applications that use the same DLLs, memory pages will not be shared among processes, thus increasing RAM usage further.

Then, Excelsior Installer already supports LZMA compression, so there won't be any significant download size reduction if you compress the executables before packaging.

Finally, with all the optimization and compression methods ever invented you won't bring the size of "Hello, World" written in Java even close to the size of "Hello, world" written in C. The Java runtime overhead would remain very big for a small utility.

That said, I really don't understand why do you care so much about disk footprint. A typical desktop computer has 100 times more hard disk space than RAM. I still have gigabytes of free disk space on my workstation, but regular slowdowns due to Firefox eating half of its 512 MB of RAM drive me crazy. It is memory usage and in some cases download size that you have to care about, not disk footprint.

Embedded systems are a totally different story of course.

Share this post


Link to post
Share on other sites

Compressing an executable file has a big disadvantage: it will not be mapped to memory by the operating system and therefore your application will need more RAM. Moreover, if you run two copies of the application, or two applications that use the same DLLs, memory pages will not be shared among processes, thus increasing RAM usage further.

Yes I am aware of this issue. However its not my intention to run my application more than once, at least I don't see any reason why a user should start two copies on the same computer.

If it would be a disadvantage for a special use-case I would still have to choice to not compress it.

By the way the "Global optimizer" has the same disadvantage. If applications would be linked only dynamically all jet-dlls could be shared every time an jet-compiled application is launched. However you also give the developer the choice here to reduce download size and enhance performance.

That said, I really don't understand why do you care so much about disk footprint. A typical desktop computer has 100 times more hard disk space than RAM. I still have gigabytes of free disk space on my workstation, but regular slowdowns due to Firefox eating half of its 512 MB of RAM drive me crazy. It is memory usage and in some cases download size that you have to care about, not disk footprint.

Well personally I would like to be able to compress jet-executable because of real use-cases:

* Deploy jet compiled demo's on an USB stick which should be able to run right out of the box. Every MB is quite expensive if you deploy hundreds of sticks.

* Add the option for companies which do not have a JRE installed to put a jet-compiled version on a shared folder so all windows-pcs could use it. And believe me it makes a difference wether you send 20mb over network or just 8mb.

* Startup time is still bad. On my Athlon-1000/512mb/60gb a jet executable has a cold startup time of about 7s and this is mostly because of disk-IO. This is about the same as jre-5 needs to start up. The packers support in-location decompression (minimal memory overhead) and decompress with about 200mb/s.

So the reason why we eventually would use JET would be to remove the burden of installing software (the JRE) just to run a mid-sized java tool. And for this task 20mb/7s are quite a lot. I could almost include a application-private jre.

I don't see why it is such a big deal to just investigate the "problem" and look whats going on. It would maybe take 10-15min for an experienced JET enineer.

The JET team has gone all downs and ups to minimize the size of compiled executables with great success, however this "optimization" would come almost for free (in terms of costs and time).

For all packers the same problem popped up, namely a segfault at "0x00000008 at address 0x71d279" - so the only interesting thing would be to find out why the packed jet-executable wants to access this adress. I am sure the packer autors would be happy to help out.

After all isn't it just a matter of coice?

If users see a sence / would like to use executable packers because of whatever, why not let them?

Thank you in advance, lg Clemens

Share this post


Link to post
Share on other sites

Compressing an executable file has a big disadvantage: it will not be mapped to memory by the operating system and therefore your application will need more RAM. Moreover, if you run two copies of the application, or two applications that use the same DLLs, memory pages will not be shared among processes, thus increasing RAM usage further.

  .  .  .

By the way the "Global optimizer" has the same disadvantage. If applications would be linked only dynamically all jet-dlls could be shared every time an jet-compiled application is launched. However you also give the developer the choice here to reduce download size and enhance performance.

This is not quite the same disadvantage. You can optimize several Java apps sharing most of their classpaths together into one EXE, adding a main() method that would dispatch control to one of those apps' main() methods accoring to the command-line arguments.

That said, I really don't understand why do you care so much about disk footprint. A typical desktop computer has 100 times more hard disk space than RAM. I still have gigabytes of free disk space on my workstation, but regular slowdowns due to Firefox eating half of its 512 MB of RAM drive me crazy. It is memory usage and in some cases download size that you have to care about, not disk footprint.

Well personally I would like to be able to compress jet-executable because of real use-cases:

* Deploy jet compiled demo's on an USB stick which should be able to run right out of the box. Every MB is quite expensive if you deploy hundreds of sticks.

This is a very good point. I have no idea why USB sticks and flash memory cards escaped my attention when I was writing my previous post...

* Add the option for companies which do not have a JRE installed to put a jet-compiled version on a shared folder so all windows-pcs could use it. And believe me it makes a difference wether you send 20mb over network or just 8mb.

OK, yet another point taken.

* Startup time is still bad. On my Athlon-1000/512mb/60gb a jet executable has a cold startup time of about 7s and this is mostly because of disk-IO. This is about the same as jre-5 needs to start up. The packers support in-location decompression (minimal memory overhead) and decompress with about 200mb/s.

Have you used the Executable Image Optimizer? It may substantially reduce the startup time, especially combined with the Global Optimizer.

So the reason why we eventually would use JET would be to remove the burden of installing software (the JRE) just to run a mid-sized java tool. And for this task 20mb/7s are quite a lot. I could almost include a application-private jre.

The disk footprint of the complete Sun JRE 5.0 is 70MB and you cannot remove much from it without violating the Sun license.

I don't see why it is such a big deal to just investigate the "problem" and look whats going on. It would maybe take 10-15min for an experienced JET enineer.

We have no idea what all those packers expect to find inside the executable, and have no resources to investigate. Most likely, this segfault is not the only problem and finding all of them  will likely take much more than 10-15 minutes.

What we could possibly do and what I think would actually work much better than reverse engineering all those packers on our side is us providing packer authors with a document describing the parts of a JET-compiled executable that must not be compressed, under a non-disclosure agreement. But again, this will take much more than 10-15 minutes and we have other commitments right now.

For all packers the same problem popped up, namely a segfault at "0x00000008 at address 0x71d279" - so the only interesting thing would be to find out why the packed jet-executable wants to access this adress.

After we figure this one out, there will likely be more "interesting things to find out". We have gone through this more than once, with our own products and with third-party software.

I am sure the packer autors would be happy to help out.

I have not heard back at all from authors of quite a few Java tools and applications whom I contacted in the hope that they might be interested in what we are doing here at Excelsior. You have to ask the packer authors whether they would be willing to help; without that, your statement is worth nothing.

If users see a sence / would like to use executable packers because of whatever, why not let them?

Sure. However, there are just a few users asking for this feature (maybe one user less, as LinuxHippy and JavaJulia post from the same IP address :) ). In particular, we have many paying customers asking for Java 6 support, that's a #1 request. I do not think we'll be able to work on anything else except Java 6 plus some features that we did not finish in time for Excelsior JET 5.0 release, schedule for next week.

Share this post


Link to post
Share on other sites
I have not heard back at all from authors of quite a few Java tools and applications whom I contacted in the hope that they might be interested in what we are doing here at Excelsior. You have to ask the packer authors whether they would be willing to help; without that, your statement is worth nothing.

Yes I already contacted the autors of PECompact and UPX in the hope somebody will investigate the problem.

I understand that there is no time left to reverse-engineer third-party packers and figure out what they are doing wrong, of course.

Sure. However, there are just a few users asking for this feature (maybe one user less, as LinuxHippy and JavaJulia post from the same IP address :) ).

Oh she's my team member, we are working together in the same project at university.

Ok it was some kind of cheating  :rolleyes:

In particular, we have many paying customers asking for Java 6 support, that's a #1 request. I do not think we'll be able to work on anything else except Java 6 plus some features that we did not finish in time for Excelsior JET 5.0 release, schedule for next week.

Oh congratulations for 5.0 :-)

Of course I Java-6 support is much more important, however I'll hope one of the packer-autors will have a look. My hope is that if they've specific questions that they'll find their answers here ;)

Thanks for your patience, lg Clemens

Share this post


Link to post
Share on other sites

After trying about 5 different packers (about two of them generated executables windows-xp refused even to load) I finally found one quite old (2002) which also works for JET.

It compressed my 12,1mb exectable down to 4,1mb and also the runtime libraries, so the total size of executable + jet-runtime-files is 8mb :-)

So if someone asks, you can mention: http://www.aspack.com/

Thanks for your patience, lg Clemens

Share this post


Link to post
Share on other sites
It compressed my 12,1mb exectable down to 4,1mb and also the runtime libraries, so the total size of executable + jet-runtime-files is 8mb :-)

My congratulations!  B)

BTW, did you check the size of the Excelsior Installer executable created for this app with JetPackII? I understand that you want to get an sfx executable not installer - I'm just curious.

The JET team has gone all downs and ups to minimize the size of compiled executables with great success

We have a nice idea about how to further descrease the size of generated executables without using compression methods. It can be done through...

                                          [To Be Continued]

B)

--ZZ Top

Share this post


Link to post
Share on other sites

The UPX-Team also found the problem why it segfaults::

This program seems to be trying to locate one its section (.jidata) by

looking at its PE header in the memory. Unfortunately it can only see the

UPX modified PE header, which results in the crash. There is nothing I can

do about this, sorry.

Maybe this can be worked-arround some time in future, as it effectivly destroys the ability to decompress in-location. I guess this is also the reson why aspack works (but needs exectly the size of the compressed executable more RAM).

lg Clemens

Share this post


Link to post
Share on other sites

The UPX-Team also found the problem why it segfaults::

This program seems to be trying to locate one its section (.jidata) by

looking at its PE header in the memory. Unfortunately it can only see the

UPX modified PE header, which results in the crash. There is nothing I can

do about this, sorry.

Maybe this can be worked-arround some time in future, as it effectivly destroys the ability to decompress in-location. I guess this is also the reson why aspack works (but needs exectly the size of the compressed executable more RAM).

Is it possible to instruct UPX or any other packer to compress executable sections selectively? 

Share this post


Link to post
Share on other sites

Is it possible to instruct UPX or any other packer to compress executable sections selectively? 

As far as I know not ... it seems aspack works just by accident because its so feature-poor ;)

Whats the reason that jet-executable access their own sections?

Thank you in advance, lg Clemens

Share this post


Link to post
Share on other sites
Whats the reason that jet-executable access their own sections?

The linker creates these sections and they are necessary for execution. Note that adding custom sections to the PE header fully complies with the PE specification and we have no idea why the UPX tool modfies them.

If the reason is encryption of executable, it makes no sense because they do not contain info about the application's code or data.

We will definitely have to implement our own best-of-the-breed exe packer. ;)

Share this post


Link to post
Share on other sites

Ok, heres another update about the whole packer story ;)

Thanks to some help in the bitsum-forums (the authors of PECompact2) I was also able to get PECompact working. (command-line switch "/Ms:No")

I did a small comparison of the two used compressors on my old Athlon1000/512Mb/Win98 and also benchmarked startup-times:

[table][tr]

[td][table][tr][td] Compressor [/td][/tr][tr][td]Uncompressed[/td][/tr][/table] AsPack[table][tr][td][/td][/tr][/table] PeCompact2 2.79 - LZMA2[table][tr][td][/td][/tr][/table][/td]

[td][table][tr][td] Size [/td][/tr][/table]12,1mb[table][tr][td]4,04mb[/td][/tr][tr][td]3,55mb[/td][/tr][/table][/td]

[td][table][tr][td] Start-Time (Cold/Warm/Warm-WinXP) [/td][/tr][/table] 8s/1s/1s[table][tr][td]3,5s/2s/1,5s[/td][/tr][tr][td]4s/2,7s/2s[/td][/tr][/table][/td][/tr][/table]

Startup of the uncompressed file is very slow if the computer has been restarted before (no disk cache) because of IO (although the computer has a 60GB HDD from 2002!), here the compressed executables startup clearly faster. On WindowsXP all executables start faster :-)

When the files are already cached, the compressed executables start up noticeable slower - PECompact a bit slower than ASPack.

I recommand using LZMA1 over LZMA2  because the LZMA2 was only 6kb smaller but needs about 300-500ms more time for decompression.

The total package was shrinked to just 7,5mb (in-folder size, 1mb pack200 archive), although some forbidden elements have been removed too (color management stuff, ...).

All in all that 3,55-4mb for a full-featured swing client are really good - and it doesn't look as fast and bloated as it really is ;)

lg Clemens

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×