Jump to content
Excelsior Forums
tanelorn

(JIT) Internal crash while bytecode compiling

Recommended Posts

Hi,

I am trying to deploy a rather large Java application (300MB of jars) using Excelsior JET. That application works properly during the initial "trial run" in JET Launcher.

The best scheme I could come with so far is the following:

- my "business classes" are compiled as an executable (about 100MB),

- third-party classes are referenced in java.class.path and compiled on demand.

The initial steps of my application works flawlessly, but then it crashes with the following:

    java.lang.InternalError: JIT ERROR: (JIT) Internal crash while bytecode compiling: out of heap space

I tried to increase the available heap by using "-heaplimit=1677721600" and "-Djet.gc.heaplimit:1677721600" in my project file, but that does not solve my problem. I am also using the "-Djet.jit.fast" option.

Could you suggest me a workaround?

Best regards.

Share this post


Link to post
Share on other sites

Hi,

Is there a verbose mode or some similar tool I could use to see which class the JIT compiler was processing while it crashed? Sadly, this information is not provided by the error message.

In this way, I could try to compile it AOT, or to replace it.

Since my first post, I tried miscellaneous GC settings such as -Djet.gc.defragment, -heaplimit=0, -Djet.gc.fatal.out.of.memory or -Djet.gc.verbose.out.of.memory. None fixed my problem or helped me gain more insight on its cause.

Best regards.

Share this post


Link to post
Share on other sites

To increase the heap for JIT compiler, use the following runtime property:

-Djet.jit.heap.size=256m

Share this post


Link to post
Share on other sites

Hi,

Thanks for your answer! I had forgotten to ask for a notification on this topic, so I did not see it. I will test this asap.

On a related note, what is the expected behaviour when the JIT compiler goes out of memory (ie, use more than 256MB in your example)?

Can it, as the JET runtime, use swap space in this situation? How can I enable this?

My application may load third-party plugins at run time, and I am concerned by the risk of out of memory errors if some of them are too large. I'd much rather prefer the application to swap in this situation than to halt with a "JIT Out of Memory" message.

Best regards.

Share this post


Link to post
Share on other sites

To increase the heap for JIT compiler, use the following runtime property:

-Djet.jit.heap.size=256m

I just tested this. Even with -Djet.jit.heap.size=1024m, JIT compilation still breaks and I get "JIT ERROR: (RT) not enough memory for JIT compiler".

I will continue to add jars to my static compilation set until the charge on the JIT compiler is minimal, but I am growing concerned with the risk of failure at runtime when dynamically-loaded plugins will get added. That may be performed by customers after shipment, where such a crash is not acceptable. So I am definitely interested by your answer to my previous message with respect to swap enabling.

Interestingly, I also tried the -Djet.jit.cache option and, after a few runs, all classes get compiled and my application behave properly. The jittemp directory is then surprinsingly small (12MB), so I have difficulties understanding why this cannot be dynamically compiled in one pass.

Best regards.

Share this post


Link to post
Share on other sites

By design of our JIT, it has explicit heap limit and may use swap but can not be adaptive as heap limit of a whole application. JITing fails if explicit heaplimit is exceeded. On the other hand, increasing the heap for JIT causes for the main application to be able allocate less memory. So there is a trade-off between how much memory could be allocated by JIT and the application.

"JIT ERROR: (RT) not enough memory for JIT compiler" is another error in compare with "JIT ERROR: (JIT) Internal crash while bytecode compiling: out of heap space". The first means that there is no physical memory to load JIT, while the second means that there is no memory to compile classes at run-time by loaded JIT. So increasing the heap for JIT to solve the second problem may cause in turn the first problem.

In your situation, I would suggest you to enable the following runtime property:

-Djet.jit.disable.resolution

In this case, JIT will compile much less classes at single JIT session, so the default JIT heaplimit setting should fit for the most cases.

Share this post


Link to post
Share on other sites
Interestingly, I also tried the -Djet.jit.cache option and, after a few runs, all classes get compiled and my application behave properly. The jittemp directory is then surprinsingly small (12MB), so I have difficulties understanding why this cannot be dynamically compiled in one pass.

Our JIT compiler is scaled down version of the main AOT compiler and its memory usage and the speed is the subject to improve for almost every JET release.  So now, it is strongly recommended to compile as many classes as possible ahead-of-time  to avoid this JIT overhead. On the other hand, you certainly cannot avoid it for plugins.

Share this post


Link to post
Share on other sites

By design of our JIT, it has explicit heap limit and may use swap but can not be adaptive as heap limit of a whole application. JITing fails if explicit heaplimit is exceeded. On the other hand, increasing the heap for JIT causes for the main application to be able allocate less memory. So there is a trade-off between how much memory could be allocated by JIT and the application.

That's very interesting. What is the default value for jet.jit.heap.size?

Is there any way to display all runtime parameters at launch time, by the way?

"JIT ERROR: (RT) not enough memory for JIT compiler" is another error in compare with "JIT ERROR: (JIT) Internal crash while bytecode compiling: out of heap space". The first means that there is no physical memory to load JIT, while the second means that there is no memory to compile classes at run-time by loaded JIT. So increasing the heap for JIT to solve the second problem may cause in turn the first problem.

Indeed, even when using small values for jet.gc.heaplimit, or a value of 0, as soon as I explicitely set jet.jit.heap.size, my application crashes with "JIT ERROR: (RT) not enough memory for JIT compiler". As an example, I still get this error when using -Djet.jit.heap.size=256m -Djet.gc.heaplimit:256m (and many other combinations of values) on a 4GB computer.

Is this to be expected?

In your situation, I would suggest you to enable the following runtime property:

-Djet.jit.disable.resolution

In this case, JIT will compile much less classes at single JIT session, so the default JIT heaplimit setting should fit for the most cases.

Yes, that did the trick for me! Using a combination of -Djet.jit.disable.resolution and -Djet.jit.cache, I can get a first slow, not crashing run, and subsequent runs have proper performances.

Thanks for your help!

Best regards.

Share this post


Link to post
Share on other sites

Our JIT compiler is scaled down version of the main AOT compiler and its memory usage and the speed is the subject to improve for almost every JET release.  So now, it is strongly recommended to compile as many classes as possible ahead-of-time  to avoid this JIT overhead. On the other hand, you certainly cannot avoid it for plugins.

I understand that now.

So we can't just switch to JET, compile a minimum set of classes and expect its JIT compiler to perform most of the work (as the SUN JIT would), isn't it? That was my initial analysis, on the basis that I only wanted to protect a minimal set of core business classes, and that compiling all my jars AOT leads to a huge 300MB DLL/EXE.

While the flag you suggested fixes my problem now, as I understand it I have no certainty that an especially taxing plugin written by one of my customers (so, not under my control and after shipment) will not generate a JIT crash. That is definitely not the same as slow-running code: my support will hate me for this  :)

Considering that the next iteration of our software will be totally plugin-based, that's a rather daunting prospect.

Would you say the main focus of Excelsior JET is and will stay AOT compilation? Or can we expect a robustness of your JIT compiler in par with the SUN JIT in the future?

No offence intended here: I just need to clearly understand those issues.

Best regards.

Share this post


Link to post
Share on other sites
What is the default value for jet.jit.heap.size?

128m

Is there any way to display all runtime parameters at launch time, by the way?

No. Some of them listed at JET User's Guide, Chapter "Application Considirations", Section "Java system properties / JET runtime properties", however there are many secret properties and there is no way to see them.

Indeed, even when using small values for jet.gc.heaplimit, or a value of 0, as soon as I explicitely set jet.jit.heap.size, my application crashes with "JIT ERROR: (RT) not enough memory for JIT compiler". As an example, I still get this error when using -Djet.jit.heap.size=256m -Djet.gc.heaplimit:256m (and many other combinations of values) on a 4GB computer.

Is this to be expected?

No, it is not expected.

Note, however, that the process' memory usage is a sum of the heap, loaded code and data of imported DLLs, its own code and data, JITed code, JIT, JIT heap, etc.

So it might happen, that even if the application itself uses not so much dynamic memory, the process may use much memory. We have seen such cases.

And it might happen that there is no memory for JIT to load, since it requires quite a large amount of continuous address space available at the point. Due to address space fragmentation, the OS may fail to provide it even if there are relatively much memory available at this time.

So, to understand if it is your case, please look at Task Manager to see how much memory your application uses just before crash with the error.  If it's far from 4 GB and there is no other processes that use much memory then probably it is a flaw in our engine. In this case, I would prefer to continue this thread via support email: java at excelsior-usa.com

However there is a (secret) workaround for the above problem. You need to enable

    -Djet.jit.do.not.unload.compiler

to force JIT to stay in the memory once it is loaded. However it causes permanent memory overhead .

So we can't just switch to JET, compile a minimum set of classes and expect its JIT compiler to perform most of the work (as the SUN JIT would), isn't it?

It depends on your meaning of "minimum set". A reasonable trade-off is to compile all classes that are always loaded by your application at any run, plus those classes that are often loaded, plus those that you wish to protect and keep the rest in the bytecode form expecting JIT compiler to cope with them.

While the flag you suggested fixes my problem now, as I understand it I have no certainty that an especially taxing plugin written by one of my customers (so, not under my control and after shipment) will not generate a JIT crash.

With this flag, there is only theoretical possibility for the compiler to lack of the memory. A typical JIT session with this flag is 1-2 classes, and there should be a very big class to make JIT failing. So you may have the certainty with the flag enabled. Note also that a plugin by users may occupy so much heap memory itself that it cannot be handled by any other JVM. Thus, in general, you have not 100% certainty with the subject at all.

Would you say the main focus of Excelsior JET is and will stay AOT compilation?

Yes, it is. And the most probably will stay. AOT is the main feature that differentiates us from other JVM implementations.

Or can we expect a robustness of your JIT compiler in par with the SUN JIT in the future?

Yes, you can expect that.

We are interested to improve and evolve any piece of our software including JIT and we have this task  on our "todo" list.

The ideal JVM for us is a real mix of AOT and JIT approaches: the application's core and platform classes should be compiled ahead-of-time, but the rest classes should be handled dynamically without significant performance penalty. I believe that both traditional JVMs such as Sun JVM and our JVM will meet at this point.

Share this post


Link to post
Share on other sites

No. Some of them listed at JET User's Guide, Chapter "Application Considirations", Section "Java system properties / JET runtime properties", however there are many secret properties and there is no way to see them.

OK, so there is no official way to see them :)

But what I was interested in is not the set of all properties, but the value of official properties... Some kind of "verbose" switch to check which heap size an application was configured with, etc.

Note, however, that the process' memory usage is a sum of the heap, loaded code and data of imported DLLs, its own code and data, JITed code, JIT, JIT heap, etc.

So it might happen, that even if the application itself uses not so much dynamic memory, the process may use much memory. We have seen such cases.

And it might happen that there is no memory for JIT to load, since it requires quite a large amount of continuous address space available at the point. Due to address space fragmentation, the OS may fail to provide it even if there are relatively much memory available at this time.

So, to understand if it is your case, please look at Task Manager to see how much memory your application uses just before crash with the error.  If it's far from 4 GB and there is no other processes that use much memory then probably it is a flaw in our engine. In this case, I would prefer to continue this thread via support email: java at excelsior-usa.com

Some investigations revealed a major memory leak in the native code part of our application, so I would not incriminate your engine. Should I see other memory-related crashes after this fix, I'll send your a more detailed report to your support address. But fine-tuning JIT behaviour with the flags you provided me already helped a lot.

However there is a (secret) workaround for the above problem. You need to enable

    -Djet.jit.do.not.unload.compiler

to force JIT to stay in the memory once it is loaded. However it causes permanent memory overhead .

That may be useful for stability purpose. But I am mostly trying to reduce my application memory consumption for now to ensure those problems will not happen again.

A reasonable trade-off is to compile all classes that are always loaded by your application at any run, plus those classes that are often loaded, plus those that you wish to protect and keep the rest in the bytecode form expecting JIT compiler to cope with them.

I would have say "hoping the JIT compiler will cope with them", but I am much more confident in it now, thanks to your explanations.

Note also that a plugin by users may occupy so much heap memory itself that it cannot be handled by any other JVM. Thus, in general, you have not 100% certainty with the subject at all.

Fair enough  ;)

Or can we expect a robustness of your JIT compiler in par with the SUN JIT in the future?

Yes, you can expect that.

We are interested to improve and evolve any piece of our software including JIT and we have this task  on our "todo" list.

That's nice to read. Moreover, considering I didn't compare the SUN and JET JIT compilers under the same memory load, I can't make statements about their relative robustness...

On an unrelated note: your forum lacks a "Solved" button initial posters could use to flag a thread when they are satisfied by your answers. I have opened quite a number of them those last weeks, and I would be glad to "close" most of them as an appreciation of your efficient support.

Best regards.

Share this post


Link to post
Share on other sites
But what I was interested in is not the set of all properties, but the value of official properties... Some kind of "verbose" switch to check which heap size an application was configured with, etc.

There is no such switch. Explicitly set properties can be viewed by means of System.getProperty(prop) call.

On an unrelated note: your forum lacks a "Solved" button initial posters could use to flag a thread when they are satisfied by your answers.

We use standard forum engine, and honestly speaking I do not know, if it is configurable in this way. Now, we consider an issue to be solved, if there is no additional questions asked by initial poster.

Share this post


Link to post
Share on other sites
That's nice to read. Moreover, considering I didn't compare the SUN and JET JIT compilers under the same memory load, I can't make statements about their relative robustness...

I would not call it "robustness". It's about memory usage of the JIT and I assure you that we pay close attention to this issue. We have already investigated the "memory bottlenecks" and now we are removing them one by one each new release of Excelsior JET. Maybe it would be better to "Kill'em all" at once but it requires lots of engineering resources and may affect the robustness.  B)

In this area, we prefer to act slower but more accurate, you know.

On an unrelated note: your forum lacks a "Solved" button initial posters could use to flag a thread when they are satisfied by your answers. I have opened quite a number of them those last weeks, and I would be glad to "close" most of them as an appreciation of your efficient support.

Feel free to post something like "thanks a lot, you helped me very much" in those topics. We will consider it as the "issue resolved" tag.

B)

Thanks for your time.

--ZZ Top

Share this post


Link to post
Share on other sites

Thanks a lot, you helped me very much  ;)

With respect to memory consumption, it's definitely understandable (and rather comforting) that you prefer to play it safe. I'll do my part by limiting my application memory allocations, too.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×