Sporadically error when casting byte[] response of HTTP call

Hello,

We have CRMScripts running at a onsite customer which do a lot of API calls, once in a while (every 1/2) days we suddenly see for about a 30 minute interval that all API requests fail (and after that work fine again without any error), they all fail with the following error:

EjScript::RunTimeException:
Called getValueAtom on ValueArray at "xx", line 205, char 29
  called from String(), declared in (unknown location)
  called from doApiGet(), declared in "xx", line 190, char 13
  called from getXXXX(), declared in "xx", line 355, char 18

The line it fails on is the following:

String rawJSONResponse = String(responseAsStream);

What we basically do in the doApiGet method is the following:

HTTP httpClass;
  
httpClass.addHeader("Content-Type", "application/json");
// add some more headers

  
NSStream responseAsStream = httpClass.getAsStream(url);

String rawJSONResponse = String(responseAsStream);
  
XMLNode jsonResult;
  
if (rawJSONResponse.isEmpty() == false && rawJSONResponse.isNull() == false)
{
    jsonResult = parseJSON2(rawJSONResponse);
}

When the errors occur we tried to debug it by enabling debug mode, but in the debug info we see that nothing strange occurs, the response is also correct.

Anyone have an idea why it would fail the convert the stream byte[] to a string sometimes? Could this be a available memory issue?

 

RE: Sporadically error when casting byte[] response of HTTP call

Hi David, 

Perhaps no one has answered because many have been, or are still, on vacation. It would be nice to know if others have experienced this as well, and if so - and most importantly, how to consistently reproduce it. I sympathize with your position, however, without knowledge how to consistantly reproduce it it's [nearly] impossible to obtain dedicated resources to look into the problem.

Best regards.

Av: Tony Yates 2. aug 2021

RE: Sporadically error when casting byte[] response of HTTP call

Hi Tony,

Thanks for response, I understand this is a hard problem, we have been unable to debug it any further, hence my post here for maybe some suggestions.

As a test, the server which this runs on has been upgraded in RAM last weekend, so we should be able to determine now if it is a resources problem.

Av: David Hollegien 2. aug 2021

RE: Sporadically error when casting byte[] response of HTTP call

Hi David, 

Could also be worth to check if http.hasError() returns some information in case it is failing in other end for some reason?

Av: Michel Krohn-Dale 2. aug 2021

RE: Sporadically error when casting byte[] response of HTTP call

Hi Michel,

Thanks for the suggestion, we tried the hasError and getErrorMessage but they don't return anything, debug info also shows that the HTTP request is succesfull.

 

Av: David Hollegien 2. aug 2021

RE: Sporadically error when casting byte[] response of HTTP call

Hello,

We still sporadically get the error:

"EjScript::RunTimeException: Called getValueAtom on ValueArray"

As mentioned we first thought it was a resources problem but that has been upgraded so we can rule that out.. Anyone have a pointer if what direction look? The response from the API is correct if we look in the debug info of the HTTP class.

Av: David Hollegien 12. aug 2021

RE: Sporadically error when casting byte[] response of HTTP call

Could you try to not use the "stream" version of HTTP get call, to see if the problem lies within there?

So something like this instead:

Byte[] responseAsBytes = httpClass.get(url);

String rawJSONResponse = String(responseAsBytes);

In addition, the first post you wrote states that the error happens on line 205. What line in the CRMScript is this?

Av: Stian Andre Olsen 12. aug 2021

RE: Sporadically error when casting byte[] response of HTTP call

Hi Stian,

Thanks, i'm going to add the bytes variant to the catch block to see if that does work.

Lines 205 referred to the line "String rawJSONResponse = String(responseAsStream);"

 

Av: David Hollegien 12. aug 2021

RE: Sporadically error when casting byte[] response of HTTP call

Hi guys, just wanted to pitch in that I'm seeing this locally when doing some testing as well.

In my case the problem happens when http.getAsStream tries to communicate with a HTTPS url that uses an untrusted certificate. If I change over to standard http.get instead, the debug info indicates "SSL certificate problem: self signed certificate in certificate chain", but with getAsStream it just crashes.

Problem, in my case, is solved by using

http.setOption("verifyPeer", "false");

Here is a script that does some more automated testing

#setLanguageLevel 3;

Vector urls;
urls.pushBack("https://www.google.com");
urls.pushBack("https://self-signed.badssl.com/");
urls.pushBack("https://expired.badssl.com/");
urls.pushBack("https://wrong.host.badssl.com/");
urls.pushBack("https://untrusted-root.badssl.com/");
urls.pushBack("https://revoked.badssl.com/");
urls.pushBack("https://pinning-test.badssl.com/");
urls.pushBack("https://httpbin.org/status/200");
urls.pushBack("https://httpbin.org/status/301");
urls.pushBack("https://httpbin.org/status/403");
urls.pushBack("https://httpbin.org/status/404");
urls.pushBack("https://httpbin.org/status/500");
urls.pushBack("https://www.youtube.com/watch?v=dQw4w9WgXcQ");

foreach (String url in urls.toStringArray()) {
  try {
    HTTP http;
    //http.setOption("verifyPeer", "false");
    //http.setOption("verifyHost", "false");
    String response = String(http.getAsStream(url));
    print(url);
    printLine("=> SUCCESS");
  } catch {
    print(url);
    printLine(" => ERROR: " + error);
  }
}

Result looks like this

https://www.google.com=> SUCCESS
https://self-signed.badssl.com/ => ERROR: EjScript::RunTimeException: Called getValueAtom on ValueArray
https://expired.badssl.com/ => ERROR: EjScript::RunTimeException: Called getValueAtom on ValueArray
https://wrong.host.badssl.com/ => ERROR: EjScript::RunTimeException: Called getValueAtom on ValueArray
https://untrusted-root.badssl.com/ => ERROR: EjScript::RunTimeException: Called getValueAtom on ValueArray
https://revoked.badssl.com/=> SUCCESS
https://pinning-test.badssl.com/=> SUCCESS
https://httpbin.org/status/200=> SUCCESS
https://httpbin.org/status/301=> SUCCESS
https://httpbin.org/status/403=> SUCCESS
https://httpbin.org/status/404=> SUCCESS
https://httpbin.org/status/500=> SUCCESS
https://www.youtube.com/watch?v=dQw4w9WgXcQ=> SUCCESS

Interrestingly doesn't seem like SuperOffice cares about revoked certificates.

By disabling verifyPeer and verifyHost all the checks will succeed.

Av: Frode Lillerud 13. aug 2021

RE: Sporadically error when casting byte[] response of HTTP call

So it looks like the .getAsStream method doesn't handle an internal http class error very well? When debugging our issue we looked at the debug info and it didn't list anything related to certifcate.

Av: David Hollegien 13. aug 2021

RE: Sporadically error when casting byte[] response of HTTP call

Found the problem.

When an exception is thrown internally from the HTTP class, it does not return correct type when using the Stream versions of the calls, and the CRMScript engine gets confused and throws an error for non-matching types. 

I have fixed this, and it will be availbale in the upcoming R11 release. In the meantime, you can use the non-stream versions.

But this probably do not fix your sporadic problems David, since this only happen when you actually get an error from the HTTP class. So I think your problem is that the HTTP calls actually fails some time. 

But with this fix (or using the non-stream versions), you can check for http.hasError(), and then be able to use http.getErrorMessage() to see what the actual error is.

Thanks for the code example Frode. 

Av: Stian Andre Olsen 13. aug 2021