HTTP class and setOption/setValue does not do the magic its supposed to

Hello !

Im trying to send data to an API and it works more than fine in POSTMAN where I am using a key as authentication.

I have imported a json collection to Postman and after import it looks like this: 

After this I try to "convert" this to CRMScript where after trial and error I'm left with this: 

HTTP export;
export.setDebugMode(true);

String usernamePassword = username + ":" + key;
String authHeader = "Basic " + encodeBase64(usernamePassword.toByteArray());

export.addHeader("Content-Type", "application/x-www-form-urlencoded");
export.addHeader("Authorization", authHeader);

export.("IMP_CSV_SEPARATOR", ",");
export.setOption("IMP_EMAIL", "umta@vitari.no");
export.setOption("IMP_PUSH_FILE", impFilePath);
export.setOption("IMP_TARGETMODE", "Replace");
export.setOption("IMP_FORMAT", "EXCEL");
export.setOption("IMP_LAYER", "sgp_sales_customers_import");
export.setOption("KEY", key);
export.setValue("IMP_IS_COMBO_OPERATION", "true");

String result =  String(export.post(urlToAPI));
print(result);
Map m = export.getResponseHeaders();

print(m.toJson());



/*
String debug = export.getDebug();
print(debug);
*/

getResponsHeaders give me: 

{"connection":"Close","content-length":"0","http/1.1 200 connection established":"","http/1.1 400 bad_request":""}

What really is the method to "convert" form-data into value in setOption or setValue when building HTTP posts ?

 

RE: HTTP class and setOption/setValue does not do the magic its supposed to

Hi Ummair,

According to documentation only following is "valid" values for setOption

  • "verifyPeer" - p_value: bool - verify the peer's SSL sertificate
  • "verifyHost" - p_value: bool - verify the certificate's name against the host
  • "timeout" - p_value: int - set maximum time the request is allowed to take
  • "followLocation" - p_value: bool - follow HTTP 3xx redirects if set to 1, default is 0
  • "username" - p_value: string - username used in authentication
  • "password" - p_value: string - password used in authentication
  • "parameters" - p_value: string - The complete parameters to post. This will replace whatever you have added using .setValue(), but allows you to specify the whole parameter to post. Useful when posting e.g. JSON structs instead of name=value pairs. NOTE: you need to prefix whatever you want to send with a dummy character (which will be removed) due to complexities in this class. See example below.
  • "authenticationMethod" - p_value - which authentication method to use. Legal values are: "basic" (default) "digest" "gssnegotiate" "ntlm" "digest_ie" "ntlm_wb" "none"
  • "parametersUTF8" - p_value: bool. From version 8.2R3. This option makes the parameters be encoded as UTF-8. Normally this is what you want, but to not break any old uses, this is an optional option.

Not sure what your API is expecting, but have you tried to use setOption("parameters", "?" + jsonString) ?

Von: Michel Krohn-Dale 28. Jan 2020

RE: HTTP class and setOption/setValue does not do the magic its supposed to

Hi Ummair, 

in Postman you're using "form-data", and in EJScript you're using "x-www-form-urlencoded". Can you get it to work in Postman using "x-www-form-urlencoded"?

Also, in Postman, you can click the "Code" link underneath the "Send" button, and see what the actual HTTP code is. You can compare that with the output of the getDebug() string from the HTTP object in CRMScript.

Von: Frode Lillerud 28. Jan 2020

RE: HTTP class and setOption/setValue does not do the magic its supposed to

Hey Frode, 

I'm supposed to send a csv/xlsx file to Nordeca's API here. I have got a JSON collection example from them, which automatically does configuration for me. I dont know what the form-data equivalent for the x-www-form-urlencoded will be.

The code on http looks like this (key is changed) :

POST /CoreServices/Import HTTP/1.1
Host: api.nordeca.com
Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW

----WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="IMP_CSV_SEPARATOR"
,

----WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="IMP_EMAIL"


----WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="IMP_PUSH_FILE"; filename="/path/to/file"
Content-Type: <Content-Type header here>
(data)

----WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="IMP_TARGETMODE"
Replace

----WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="IMP_FORMAT"
EXCEL

----WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="IMP_LAYER"
sales_customers_import

----WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="KEY"
156c0dce-aa5yy1-4359-a9c9-d0c4470aui53b8

----WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="IMP_IS_COMBO_OPERATION"
true

----WebKitFormBoundary7MA4YWxkTrZu0gW

Could it be that i can send the HTTP from  in the setOption("parameters", "?" + "httpBody" );  ? 

Also this is the json collection I have as a starting point for postman: 

{
	"info": {
		"_postman_id": "528bd183-7166-4fa0-9941-d20f8cdcd450",
		"name": "SGP",
		"schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json"
	},
	"item": [
		{
			"name": "sgp_sales_activities_import",
			"request": {
				"method": "POST",
				"header": [
					{
						"key": "Content-Type",
						"name": "Content-Type",
						"type": "text",
						"value": "application/x-www-form-urlencoded"
					}
				],
				"body": {
					"mode": "formdata",
					"formdata": [
						{
							"key": "IMP_EMAIL",
							"value": "",
							"type": "text"
						},
						{
							"key": "IMP_PUSH_FILE",
							"description": "Needs to be the actual file, not the file name.",
							"type": "file",
							"src": ""
						},
						{
							"key": "IMP_TARGETMODE",
							"value": "Replace",
							"type": "text"
						},
						{
							"key": "IMP_FORMAT",
							"value": "EXCEL",
							"type": "text"
						},
						{
							"key": "IMP_LAYER",
							"value": "sgp_sales_activities_import",
							"type": "text"
						},
						{
							"key": "KEY",
							"value": "",
							"type": "text"
						}
					]
				},
				"url": {
					"raw": "https://api.nordeca.com/CoreServices/Import",
					"protocol": "https",
					"host": [
						"api",
						"nordeca",
						"com"
					],
					"path": [
						"CoreServices",
						"Import"
					]
				}
			},
			"response": []
		},
		{
			"name": "sgp_sales_customers_import",
			"request": {
				"method": "POST",
				"header": [
					{
						"key": "Content-Type",
						"name": "Content-Type",
						"value": "application/x-www-form-urlencoded",
						"type": "text"
					}
				],
				"body": {
					"mode": "formdata",
					"formdata": [
						{
							"key": "IMP_CSV_SEPARATOR",
							"value": ",",
							"type": "text"
						},
						{
							"key": "IMP_EMAIL",
							"value": "",
							"description": "Mail for infromation",
							"type": "text"
						},
						{
							"key": "IMP_PUSH_FILE",
							"description": "Needs to be the actual file, not the file name.",
							"type": "file",
							"src": ""
						},
						{
							"key": "IMP_TARGETMODE",
							"value": "Replace",
							"type": "text"
						},
						{
							"key": "IMP_FORMAT",
							"value": "EXCEL",
							"type": "text"
						},
						{
							"key": "IMP_LAYER",
							"value": "sgp_sales_customers_import",
							"type": "text"
						},
						{
							"key": "KEY",
							"value": "",
							"type": "text"
						},
						{
							"key": "IMP_IS_COMBO_OPERATION",
							"value": "true",
							"type": "text"
						}
					]
				},
				"url": {
					"raw": "https://api.nordeca.com/CoreServices/Import",
					"protocol": "https",
					"host": [
						"api",
						"nordeca",
						"com"
					],
					"path": [
						"CoreServices",
						"Import"
					]
				}
			},
			"response": []
		},
		{
			"name": "sgp_sales_employees_import",
			"request": {
				"method": "POST",
				"header": [
					{
						"key": "Content-Type",
						"name": "Content-Type",
						"type": "text",
						"value": "application/x-www-form-urlencoded"
					}
				],
				"body": {
					"mode": "formdata",
					"formdata": [
						{
							"key": "IMP_CSV_SEPARATOR",
							"value": ",",
							"description": "CSV separator, only used for CSV import",
							"type": "text"
						},
						{
							"key": "IMP_EMAIL",
							"value": "",
							"description": "Report email",
							"type": "text"
						},
						{
							"key": "IMP_PUSH_FILE",
							"description": "Actual file.",
							"type": "file",
							"src": ""
						},
						{
							"key": "IMP_TARGETMODE",
							"value": "Replace",
							"description": "Replace, Append, Update_Add, Update_only, Delete",
							"type": "text"
						},
						{
							"key": "IMP_FORMAT",
							"value": "EXCEL",
							"description": "Format CSV/EXCEL",
							"type": "text"
						},
						{
							"key": "IMP_LAYER",
							"value": "sgp_sales_employees_import",
							"description": "LayerCodename",
							"type": "text"
						},
						{
							"key": "KEY",
							"value": "",
							"description": "Access Key",
							"type": "text"
						},
						{
							"key": "IMP_IS_COMBO_OPERATION",
							"value": "true",
							"description": "To run geocoding as part of the import",
							"type": "text"
						}
					]
				},
				"url": {
					"raw": "https://api.nordeca.com/CoreServices/Import",
					"protocol": "https",
					"host": [
						"api",
						"nordeca",
						"com"
					],
					"path": [
						"CoreServices",
						"Import"
					]
				}
			},
			"response": []
		},
		{
			"name": "sgp_sales_turnover_import",
			"request": {
				"method": "POST",
				"header": [
					{
						"key": "Content-Type",
						"name": "Content-Type",
						"type": "text",
						"value": "application/x-www-form-urlencoded"
					}
				],
				"body": {
					"mode": "formdata",
					"formdata": [
						{
							"key": "IMP_CSV_SEPARATOR",
							"value": ",",
							"description": "CSV separator, only used for CSV import",
							"type": "text"
						},
						{
							"key": "IMP_EMAIL",
							"value": "",
							"description": "Report email",
							"type": "text"
						},
						{
							"key": "IMP_PUSH_FILE",
							"description": "Actual file.",
							"type": "file",
							"src": ""
						},
						{
							"key": "IMP_TARGETMODE",
							"value": "Replace",
							"description": "Replace, Append, Update_Add, Update_only, Delete",
							"type": "text"
						},
						{
							"key": "IMP_FORMAT",
							"value": "EXCEL",
							"description": "Format CSV/EXCEL",
							"type": "text"
						},
						{
							"key": "IMP_LAYER",
							"value": "sgp_sales_turnover_import",
							"description": "LayerCodename",
							"type": "text"
						},
						{
							"key": "KEY",
							"value": "",
							"description": "Access Key",
							"type": "text"
						},
						{
							"key": "IMP_IS_COMBO_OPERATION",
							"value": "true",
							"description": "To run geocoding as part of the import",
							"type": "text"
						}
					]
				},
				"url": {
					"raw": "https://api.nordeca.com/CoreServices/Import",
					"protocol": "https",
					"host": [
						"api",
						"nordeca",
						"com"
					],
					"path": [
						"CoreServices",
						"Import"
					]
				}
			},
			"response": []
		},
		{
			"name": "sgp_sales_orders_import",
			"request": {
				"method": "POST",
				"header": [
					{
						"key": "Content-Type",
						"name": "Content-Type",
						"type": "text",
						"value": "application/x-www-form-urlencoded"
					}
				],
				"body": {
					"mode": "formdata",
					"formdata": [
						{
							"key": "IMP_CSV_SEPARATOR",
							"value": ",",
							"description": "CSV separator, only used for CSV import",
							"type": "text"
						},
						{
							"key": "IMP_EMAIL",
							"value": "",
							"description": "Report email",
							"type": "text"
						},
						{
							"key": "IMP_PUSH_FILE",
							"description": "Actual file.",
							"type": "file",
							"src": ""
						},
						{
							"key": "IMP_TARGETMODE",
							"value": "Replace",
							"description": "Replace, Append, Update_Add, Update_only, Delete",
							"type": "text"
						},
						{
							"key": "IMP_FORMAT",
							"value": "EXCEL",
							"description": "Format CSV/EXCEL",
							"type": "text"
						},
						{
							"key": "IMP_LAYER",
							"value": "sgp_sales_orders_import",
							"description": "LayerCodename",
							"type": "text"
						},
						{
							"key": "KEY",
							"value": "",
							"description": "Access Key",
							"type": "text"
						},
						{
							"key": "IMP_IS_COMBO_OPERATION",
							"value": "true",
							"description": "To run geocoding as part of the import",
							"type": "text"
						}
					]
				},
				"url": {
					"raw": "https://api.nordeca.com/CoreServices/Import",
					"protocol": "https",
					"host": [
						"api",
						"nordeca",
						"com"
					],
					"path": [
						"CoreServices",
						"Import"
					]
				}
			},
			"response": []
		},
		{
			"name": "template import Request",
			"request": {
				"method": "POST",
				"header": [
					{
						"key": "Content-Type",
						"name": "Content-Type",
						"value": "application/x-www-form-urlencoded",
						"type": "text"
					}
				],
				"body": {
					"mode": "formdata",
					"formdata": [
						{
							"key": "IMP_EMAIL",
							"value": "",
							"description": "Report email",
							"type": "text"
						},
						{
							"key": "IMP_PUSH_FILE",
							"description": "Actual file.",
							"type": "file",
							"src": []
						},
						{
							"key": "IMP_TARGETMODE",
							"value": "Replace",
							"description": "Replace, Append, Update_Add, Update_only, Delete",
							"type": "text"
						},
						{
							"key": "IMP_FORMAT",
							"value": "EXCEL",
							"description": "Format CSV/EXCEL",
							"type": "text"
						},
						{
							"key": "IMP_LAYER",
							"value": "",
							"description": "LayerCodename",
							"type": "text"
						},
						{
							"key": "KEY",
							"value": "",
							"description": "Access Key",
							"type": "text"
						},
						{
							"key": "IMP_IS_COMBO_OPERATION",
							"value": "true",
							"description": "To run geocoding as part of the import",
							"type": "text"
						}
					]
				},
				"url": {
					"raw": "https://api.nordeca.com/CoreServices/Import",
					"protocol": "https",
					"host": [
						"api",
						"nordeca",
						"com"
					],
					"path": [
						"CoreServices",
						"Import"
					]
				}
			},
			"response": []
		}
	],
	"event": [
		{
			"listen": "prerequest",
			"script": {
				"id": "389d4c26-8425-4b90-9cf7-7065f678ca37",
				"type": "text/javascript",
				"exec": [
					""
				]
			}
		},
		{
			"listen": "test",
			"script": {
				"id": "c0588337-0b48-48c5-86f9-6e290fdbe0b8",
				"type": "text/javascript",
				"exec": [
					""
				]
			}
		}
	]
}

 

 

Von: Ummair Tahir 28. Jan 2020

RE: HTTP class and setOption/setValue does not do the magic its supposed to

You might need to build up the content manually, with the boundaries etc. See this Stack overflow thread for hints on how it can be done. https://stackoverflow.com/a/23207962

Von: Frode Lillerud 28. Jan 2020

RE: HTTP class and setOption/setValue does not do the magic its supposed to

Would it forinstance be better to just write code in another language (C# or Java) instead of using time trying to convert this to CRMScript ?

Von: Ummair Tahir 29. Jan 2020

RE: HTTP class and setOption/setValue does not do the magic its supposed to

Would it forinstance be better to just write code in another language (C# or Java) instead of using time trying to convert this to CRMScript ?

Von: Ummair Tahir 29. Jan 2020

RE: HTTP class and setOption/setValue does not do the magic its supposed to

This is a very interesting challenge :)

As Frode points out, your endpoint probably only works with multipart/form-data. I have not seen many API's using this.

I see that our underlying HTTPConnection class actually supports posting multipart data, but this functionality has not been exposed to CRMScript. We could probably put it on our backlog, but that is not going to solve your problem right now.

I played around with this a little bit, following Frodes suggestion of creating the multipart body yourself. Most of this is quite easy, however files are a bit tricky. The common way to post files in a multipart form is to send the binary data right inside the body. At least, this is what browsers have been doing since we implemented support for this back in 1998... (Netscape was the first browser to support files in form posts I believe). The problem with this, is that you are specifying the parameters using a String in CRMScript, and consequently you cannot add a binary file containing a 0 somewhere, as it will terminate the String. 

Our own parsing of attachments only support binary data, but I am thinking that a more professional library perhaps supports encoding the attachment, e.g. in base64. In that case, you should be able to create a multipart body containing both parameters and files. I tried with the one at httpbin.org, and at least there it worked. You should try with your own API endpoint and see if it will accept a multipart post with base64 files.

Here is my test code, which at least works towards httpbin.org. Without the attachment, it also worked doing a newTicket post towards our Customer Center. It is hardcoded, but you can easily create a couple of small functions to generalize it:

#setLanguageLevel 3;

String delimiter = "--------------------------087790448929171267639147";

String postData = 
  "--" + delimiter + "\r\n" +
  "Content-Disposition: form-data; name=\"action\"\r\n" +
  "\r\n" + 
  "newTicket\r\n" +
  "--" + delimiter + "\r\n" +
  "Content-Disposition: form-data; name=\"custName\"\r\n" +
  "\r\n" + 
  "Sverre Hjelm\r\n" +
  "--" + delimiter + "\r\n" +
  "Content-Disposition: form-data; name=\"custEmail\"\r\n" +
  "\r\n" + 
  "sverre.hjelm@superoffice.com\r\n" +
  "--" + delimiter + "\r\n" +
  "Content-Disposition: form-data; name=\"category\"\r\n" +
  "\r\n" + 
  "1\r\n" +
  "--" + delimiter + "\r\n" +
  "Content-Disposition: form-data; name=\"title\"\r\n" +
  "\r\n" + 
  "abcd\r\n" +
  "--" + delimiter + "\r\n" +
  "Content-Disposition: form-data; name=\"message\"\r\n" +
  "\r\n" + 
  "abcd\r\n" +
  "--" + delimiter + "\r\n" +
  "Content-Disposition: form-data; name=\"attachment\";filename=\"test.txt\"\r\n" +
  "Content-Transfer-Encoding: base64\r\n" +
  "Content-Type: image/png\r\n" +
  "\r\n" + 
  "iVBORw0KGgoAAAANSUhEUgAAAQAAAAEACAYAAABccqhmAAADO0lEQVR42u3dMYoTYRiA4TchiEhYxBN4DMHGcgvP4FU8iqVHsPck1ouFSFjCEjIWJrjFFsKuJpP/eSAwmSIJH5k3+af5F1VTVV1V6+rZ4XjZn/PH4+Pz+5YPnHvI88PjMdbVKqp+Vvsz/Wzbw+Pc7avNGc/xvttq98Sv+W0x1bvqyxNcnMC8fDz+erv4YUBLIwABAAYNwNoYYNwAuKsOlgCAAAACAAgAIACAAAACAAgAIACAAAACAAgAMJM­AiAAMHIArYwBLAEAAAAEABAAQAEAAAAEABAAQAEAAgLk59aYg++p789ieGf6Fl51wc95TB+Cmetvvvc9hRJ+q61EDsKtuFrXxPWBEU23dAwAEABAAQAAAAQAEABAAQAAAAQAEABAAQAAAAQAEABAAQAAAAQAEAAQAEABAAAABAAQAuPgArIwBxg3A2hjAEgAQAEAAAAEABAAQAEAAAAEABAAQAEAAAAEABAAQAEAAAAEABAAQAEAAAAEABAAQAEAAAAEABAAQAEAAAAEABAAQAEAAAAEABAAQAEAAAAE­AAQAEABAAQAAAAQAEABAAQACASwnArTHAuAG4MwawBAAEABAAQAAAAQAEABAAQAAAAQAEABAAQAAAAQAEABAAQAAAAQAEABAAQAAAAQAEABAAQAAAAQAEABAAQAAAAQAEABAAQAAAAQAEABAAQABAAAABAAQAEABAAAABAAQAEABAAIDZB+CHMYB/AIAAAAIACAAgAIAAAAIACAAgAIAAAAIACAAgAIAAAAIACAAgAIAAAAIACAAgAIAAAAIACAAgAIAAAAIACAAgAIAAAAIACAAgAIAAAAIACAAIACA­AgAAAAgAIAHDRAdgYA4wbgJ0xgCUAIACAAAACAAgAIACAAAACAAgAIACAAAACAAgAIACAAAACAAgAIACAAAACAAgAIACAAAACAAgAIACAAAACAAgAIADA31id+P1fVZ8nW5QzrjcjB+BF9d53ACwBAAEABAAQAEAAAAEABAAQAODxAdgaA4wbgDtjAEsAQAAAAQAEABAAQAAAAQAEABAAQAAAAQAEABAAQAAAAQAEABAAQAAAAQD+u8VUr6sPRgHD+foLVUknAXHGDuIAAAAASUVORK5CYII=\r\n" +
  "--" + delimiter + "\r\n" +
  "Content-Disposition: form-data; name=\"ok\"\r\n" +
  "\r\n" + 
  "Ok\r\n" +
  "--" + delimiter + "--" + "\r\n";

HTTP h;
h.setOption("parameters", "?" + postData);
h.addHeader("Content-Type", "multipart/form-data; boundary=" + delimiter);
h.addHeader("Content-Length", postData.getLength().toString());
printLine(String(h.post("http://httpbin.org/anything")));

Sverre

Von: Sverre Hjelm 29. Jan 2020

RE: HTTP class and setOption/setValue does not do the magic its supposed to

Hey Sverre !

Sadly I didnt get notification on this forum post and played with C# to solve this. 

I will try out your suggestion. 

Thank you !

Von: Ummair Tahir 5. Feb 2020

RE: HTTP class and setOption/setValue does not do the magic its supposed to

Hello, 

@Sverre You wrote this in a previous post:
"I see that our underlying HTTPConnection class actually supports posting multipart data, but this functionality has not been exposed to CRMScript. We could probably put it on our backlog, but that is not going to solve your problem right now."  

Any idea on when/if this will ever get implemented? I am working on a solution where this is relevant, and I have to decide if I should start working on what you did (Create it manually) or wait for it to be 'standard' in Online. 
If its undecided then I would like to make it a wish to be implemented 'down the line' :) 

//Eivind

Von: Eivind Johan Fasting 17. Feb 2020