Wie programmiere ich in Grimport?

Bitten Sie uns um Hilfe bei der Programmierung in Grimport TRANSLATE THIS PAGE WITHOUT TOUCHING THE CODES
Summary
  1. Grimport Language Documentation
    1. Introduction
    2. What is Grimport?
    3. Why use Grimport?
    4. Paradigms of language
  2. Installation of Grimport & Other Software
    1. How to get Grimport
    2. Other software to install
    3. Practicing software
  3. Variables & Syntax
    1. What is a variable?
    2. Escape sequence
    3. Arithmetic operators
    4. Relational operators
    5. Logical operators
    6. Conditional operators
    7. Lists
    8. Handling null values
    9. Scope rules
    10. Functions
    11. Optional arguments
    12. Default variables
    13. Comments
  4. Control Structures
    1. Conditional structures
    2. Looping structures
    3. Methods for iterating
    4. The break & continue statements
  5. Extract data from a page
    1. Scraping data
    2. Data cleaning
  6. Export Filtered Data
    1. File export (CSV, Excel)
    2. Export with a CMS
    3. Export to remote server
  7. Programming with Grimport Script Editor
    1. Introducing Grimport Script Editor
    2. Using Grimport Script Editor
  8. Crawler and Logs
  9. GRS, GRC & GRL
  10. Folder Organization
  11. Launch Options
  12. Cache
  13. Mistakes & Errors
    1. Finding the cause of an error
    2. Common mistakes/errors at the beginning
    3. Error codes (500, 503, etc.)
    4. Java
    5. Other
  14. Connected services
    1. The Cloud
    2. Cloud Endpoint
    3. Third-party APIs with Grimport
  15. To Go Further
    1. Parallelism
    2. Graphical elements in the Wizard
    3. Classes / Objects
    4. Java & Groovy inheritance
    5. Import a script
    6. Good coding practices

The Cloud

The Cloud allows you to run your scripts on idIA Tech's servers rather than on your own PC.
This allows you to preserve your resources (CPU, RAM, Web traffic) but also to avoid automatic restarts in the middle of the crawling.
You will be able to access your old crawls and remotely monitor your crawls as if they were running on your machine. You can also schedule your crawls at regular intervals thanks to a CRON.

Here are the different steps to run your scripts with the cloud:

  1. Copy your cloud code from your Idia Tech account into Grimport Script Editor parameters

  2. Choose the scipt to run, then go to the Cloud tab

  3. To launch the crawling:
    • You can click on "Launch in the Cloud"
    • Or click on "Schedule regularly in the Cloud" to schedule your crawling

 

It is possible to change the behavior of the script in normal mode and in cloud mode with the isCloud() function.

The function returns true if the script is executed on the Cloud, false if not.

Example:

if(isCould())
{ 

	functionIfCloud() 
} 
else
{  
	functionIfOnNormalPost() 
}

 

Cloud Endpoint

It is possible to launch a script manually from Grimport Script Editor or to program a CRON. But you can also proceed in PUSH, i.e. dynamic launches on demand thanks to an endpoint.

URL of the endpoint: https://www.idia-tech.com/endpoint_script.php

This endpoint is very easy to use, you can call it by any of these 3 methods: GET, POST, JSON.

 

Endpoint parameters

  • server_id: Identifier of the Cloud server on which you will run the script. Ask your idIA Tech contact for it.
  • pass: Your API password. Ask your idIA Tech contact. This is not your Cloud ID, you must be granted API access.
  • link: (optionnal) Connection link. It allows to identify the interfaced site on which the script will act. For example, it is the connection link of the Prestashop module similar to this https://site-prestashop.com/module/megaimporter/communication?clef=19255624. If you act with a local script use the link LOCAL:-1
  • script_id: The unique identifier of the script. You get it in Grimport Script Editor in the Options tab > Copy Unique Script ID.
  • options: (optionnal) The options for launching the script. For example: nodisplay,requestReporting. All available options are accessible in Grimport Script Editor when you click on the gear (Options for the launcher). Tooltips tell you what each option does.
  • designation: Allows you to give a specific name to the crawling.

 

Method GET

This is the easiest to use, it is a link with arguments and values. Don't forget to URL-encode the argument values to avoid syntax conflicts.

For example, you can call this page: https://www.idia-tech.com/endpoint_script.php?pass=dfsqe8eEKd4zUDs5sq83&server_id=12&link=LOCAL:-1&script_id=3372

curl -X GET https://www.idia-tech.com/endpoint_script.php?pass=PASS&server_id=ID_SERVER&link=LINK_SITE&script_id=ID_SCRIPT&options=OPTIONS

<?php

$response = file_get_contents("https://www.idia-tech.com/endpoint_script.php?pass=PASS&server_id=ID_SERVER&link=LINK_SITE&script_id=ID_SCRIPT&options=OPTIONS");

if($response == 'OK') echo 'Success';
else echo 'Error:'.$response;

URL obj = new URL("https://www.idia-tech.com/endpoint_script.php?pass=PASS&server_id=ID_SERVER&link=LINK_SITE&script_id=ID_SCRIPT&options=OPTIONS");
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
con.setRequestMethod("GET");
int responseCode = con.getResponseCode();
System.out.println("GET Response Code : " + responseCode);

BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();

while ((inputLine = in.readLine()) != null) 
{
	response.append(inputLine);
}
in.close();

if(response.toString().equals("OK")) System.out.println("Success");
else System.out.println("Error: "+response.toString());

response = getPage("https://www.idia-tech.com/endpoint_script.php?pass=PASS&server_id=ID_SERVER&link=LINK_SITE&script_id=ID_SCRIPT&options=OPTIONS")

if(equals(response, "OK")) console("Success")
else console("Error: "+response)

 

Method POST

This is a classic POST request with body arguments. It is the same type as those sent by HTML forms. The parameters of the endpoints are keys/values of the fields of the POST request.

curl -X POST -d 'pass=PASS&server_id=ID_SERVER&link=LINK_SITE&script_id=ID_SCRIPT&options=OPTIONS' https://www.idia-tech.com/endpoint_script.php

<?php

$url = 'https://www.idia-tech.com/endpoint_script.php';
$data = array(
	"pass" => "PASS",
	"server_id" => "ID_SERVER",
	"link" => "LINK_SITE",
	"script_id" => "ID_SCRIPT",
	"options" => "OPTIONS"
);

$options = array(
    'http' => array(
        'header'  => "Content-type: application/x-www-form-urlencoded\r\n",
        'method'  => 'POST',
        'content' => http_build_query($data)
    )
);
$context  = stream_context_create($options);
$response = file_get_contents($url, false, $context);

if($response == 'OK') echo 'Success';
else echo 'Error:'.$response;

URL url = new URL("https://www.idia-tech.com/endpoint_script.php");
URLConnection con = url.openConnection();
HttpURLConnection http = (HttpURLConnection)con;
http.setRequestMethod("POST");
http.setDoOutput(true);

Map<String,String> arguments = new HashMap<>();
arguments.put("pass", "PASS");
arguments.put("server_id", "ID_SERVER");
arguments.put("link", "LINK_SITE");
arguments.put("script_id", "ID_SCRIPT");
arguments.put("options", "OPTIONS");
StringJoiner sj = new StringJoiner("&");
for(Map.Entry<String,String> entry : arguments.entrySet())
    sj.add(URLEncoder.encode(entry.getKey(), "UTF-8") + "=" 
         + URLEncoder.encode(entry.getValue(), "UTF-8"));
byte[] out = sj.toString().getBytes(StandardCharsets.UTF_8);
int length = out.length;

http.setFixedLengthStreamingMode(length);
http.setRequestProperty("Content-Type", "application/x-www-form-urlencoded; charset=UTF-8");
http.connect();
try(OutputStream os = http.getOutputStream()) {
    os.write(out);
}

int responseCode = http.getResponseCode();
System.out.println("GET Response Code : " + responseCode);

BufferedReader in = new BufferedReader(new InputStreamReader(http.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();

while ((inputLine = in.readLine()) != null) 
{
	response.append(inputLine);
}
in.close();

if(response.toString().equals("OK")) System.out.println("Success");
else System.out.println("Error: "+response.toString());

response = post("https://www.idia-tech.com/endpoint_script.php", [
	"pass": "PASS",
	"server_id": "ID_SERVER",
	"link": "LINK_SITE",
	"script_id": "ID_SCRIPT",
	"options": "OPTIONS"
])

if(equals(response, "OK")) console("Success")
else console("Error: "+response)

 

Method JSON

Here we also send a POST request with a forced body that contains the JSON format of an array with the endpoints arguments in this format:

{
     "server_id": "ID_SERVER",
     "pass": "PASS",
     "link": "LINK_SITE",
     "script_id": "ID_SCRIPT",
     "options": "OPTIONS"
}

Usually the content type is application/json on this type of request, it is not mandatory here.

curl -X POST https://www.idia-tech.com/endpoint_script.php
   -H 'Content-Type: application/json'
   -d '{  "server_id": "ID_SERVER",  "pass": "PASS",  "link": "LINK_SITE",  "script_id": "ID_SCRIPT",  "options": "OPTIONS"}'

<?php

$url = 'https://www.idia-tech.com/endpoint_script.php';
$data = array(
	"pass" => "PASS",
	"server_id" => "ID_SERVER",
	"link" => "LINK_SITE",
	"script_id" => "ID_SCRIPT",
	"options" => "OPTIONS"
);

$options = array(
    'http' => array(
    	'header'=>  "Content-Type: application/json\r\n" .
              	    "Accept: application/json\r\n",
        'method'  => 'POST',
        'content' => json_encode($data)
    )
);
$context  = stream_context_create($options);
$response = file_get_contents($url, false, $context);

if($response == 'OK') echo 'Success';
else echo 'Error:'.$response;

URL url = new URL("https://www.idia-tech.com/endpoint_script.php");
URLConnection con = url.openConnection();
HttpURLConnection http = (HttpURLConnection)con;
http.setRequestMethod("POST");
http.setDoOutput(true);

byte[] out = "{  \"server_id\": \"ID_SERVER\",  \"pass\": \"PASS\",  \"link\": \"LINK_SITE\",  \"script_id\": \"ID_SCRIPT\",  \"options\": \"OPTIONS\"}" .getBytes(StandardCharsets.UTF_8);
int length = out.length;

http.setFixedLengthStreamingMode(length);
http.setRequestProperty("Content-Type", "application/json; charset=UTF-8");
http.connect();
try(OutputStream os = http.getOutputStream()) {
    os.write(out);
}

int responseCode = http.getResponseCode();
System.out.println("GET Response Code : " + responseCode);

BufferedReader in = new BufferedReader(new InputStreamReader(http.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();

while ((inputLine = in.readLine()) != null) 
{
	response.append(inputLine);
}
in.close();

if(response.toString().equals("OK")) System.out.println("Success");
else System.out.println("Error: "+response.toString());

response = postBody("https://www.idia-tech.com/endpoint_script.php", jsonEncode([
	"pass": "PASS",
	"server_id": "ID_SERVER",
	"link": "LINK_SITE",
	"script_id": "ID_SCRIPT",
	"options": "OPTIONS"
]))

if(equals(response, "OK")) console("Success")
else console("Error: "+response)

 

Response of the API

The API's answer is just as simple. The request will return the text OK if the script was run (HTTP code 200), if an error occurs, another HTTP code than 200 is issued and the error is displayed.

If you want a JSON response format {"Success": false, "Error": "Here the error explained"}, set the json_response argument to 1 (...&json_response=1).

curl -X GET https://www.idia-tech.com/endpoint_script.php?pass=PASS&server_id=ID_SERVER&link=LINK_SITE&script_id=ID_SCRIPT&options=OPTIONS&json_response=1

<?php

$response = file_get_contents("https://www.idia-tech.com/endpoint_script.php?pass=PASS&server_id=ID_SERVER&link=LINK_SITE&script_id=ID_SCRIPT&options=OPTIONS&json_response=1");

URL obj = new URL("https://www.idia-tech.com/endpoint_script.php?pass=PASS&server_id=ID_SERVER&link=LINK_SITE&script_id=ID_SCRIPT&options=OPTIONS&json_response=1");
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
con.setRequestMethod("GET");
int responseCode = con.getResponseCode();
System.out.println("GET Response Code : " + responseCode);

BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();

while ((inputLine = in.readLine()) != null) 
{
	response.append(inputLine);
}
in.close();

response = getPage("https://www.idia-tech.com/endpoint_script.php?pass=PASS&server_id=ID_SERVER&link=LINK_SITE&script_id=ID_SCRIPT&options=OPTIONS&json_response=1")

 

Reuse the variables transmitted by the endpoint in your script

You can use 3 variables in Grimport script, _GET, _POST and _JSON. These variables are maps of arguments. For example, _GET allows you to access to all GET arguments sent by the API call of the endpoint.

With get(_GET, "link") you access the link variable passed in GET. Even if non-standard API parameters are passed, you can access them.

Use if(arrayContains(global, "_GET", false)) to know if you are in an appropriate context to use these variables.

Another method is available, you can use the globals argument of the options. For example, if you put in options this text: globals={"varName":"value"}, you will instantiate at the beginning of the script a variable varName with the value "value".

 

Return the crawling response

Normally, the endpoint quickly terminates by reporting that a request to add a crawling to the cloud has been successful.

You can use the synchronous argument to change this behaviour (...&synchronous=1). The call to the endpoint runs synchronously. It only ends when the setEndpointResponse function is called in the script and a response has been returned for the endpoint. The call will stop after 5 minutes anyway. The crawling should therefore take less time.

Example script:

response = [
	"id_serveur": getCloudIdServer(),
	"nb_task_on_cloud": count(getAllRunningScripts())
]
setEndpointResponse(response, "json")

Example of a call to the endpoint with the GET method:

curl -X GET https://www.idia-tech.com/endpoint_script.php?pass=PASS&server_id=ID_SERVER&link=LINK_SITE&script_id=ID_SCRIPT&options=OPTIONS&synchronous=1

<?php

$response = file_get_contents("https://www.idia-tech.com/endpoint_script.php?pass=PASS&server_id=ID_SERVER&link=LINK_SITE&script_id=ID_SCRIPT&options=OPTIONS&synchronous=1");

$mapResponse=json_decode($response);
echo 'Number of current tasks on the Cloud serveur: '.$mapResponse["nb_task_on_cloud"];

URL obj = new URL("https://www.idia-tech.com/endpoint_script.php?pass=PASS&server_id=ID_SERVER&link=LINK_SITE&script_id=ID_SCRIPT&options=OPTIONS&synchronous=1");
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
con.setRequestMethod("GET");
int responseCode = con.getResponseCode();
System.out.println("GET Response Code : " + responseCode);

BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();

while ((inputLine = in.readLine()) != null) 
{
	response.append(inputLine);
}
in.close();

try{
	JSONParser parser = new JSONParser();
	Object obj = parser.parse(response.toString());
	if(obj!=null && obj instanceof JSONObject)
	{
		Map<String, Object> map=new Map<String, Object> ();
		Iterator<String> keysItr = ((JSONObject)obj).keySet().iterator();
	    while(keysItr.hasNext()) {
	        String key = keysItr.next();
	        Object value = object.get(key);
	        map.put(key, value);
		}
	}
	
	System.out.println("Number of current tasks on the Cloud serveur: "+map.get("nb_task_on_cloud"));
}catch(ParseException pe){
	pe.printStackTrace();
}

response = getPage("https://www.idia-tech.com/endpoint_script.php?pass=PASS&server_id=ID_SERVER&link=LINK_SITE&script_id=ID_SCRIPT&options=OPTIONS&synchronous=1")

mapResponse = jsonDecode(response)
console("Number of current tasks on the Cloud serveur: "+get(mapResponse, "nb_task_on_cloud"))

 

Third-party APIs

 

API definition

What is an API?
An Application Programming Interface, also called API, is a software interface that allows one software or service to be "connected" to another software or service in order to exchange data and functionality.

 

What are third-party APIs?
Third-party APIs are APIs that are provided by third parties, usually companies like Facebook, Twitter or Google, that allow access to their data and functionality in order to use them on its site (e.g. using Google Translate to translate your texts).

Grimport is full of pre-integrated APIs, which will make your developments much easier.

 

Third-party APIs with grimport

In this section, we'll look at some third-party APIs you can use with Grimport.

Third-party translation APIs

In Grimport there are functions to translate text:

For example, the translateGoogle() function allows you to translate text with Google Translate.
Normally all translation services are present in Grimport:

Other third-party APIs

There are many other functions:

  • firefox() allows you to easily interact with Firefox. Grimport and Firefox will then be synchronized.

  • antiCaptcha() allows you to solve a captcha and return the solution or a key to pass the captcha.

  • proxyChange() allows you to use rotating proxy systems. This function is compatible with Tor, CyberGhost, NordVPN and OpenVPN.

  • getNbGoogleResults() returns the number of results found by Google. It allows to have a good estimation of the number of pages of a website.

  • serpGoogle() allows to get the results of a Google search.

  • interact() allows you to simulate the keyboard and mouse movements.

 

Third-party APIs integration

All these third party APIs are pre-integrated in the Grimport code but it is possible to add them by yourself.

To add one, you will have to consult the API documentation. It describes how to connect it efficiently wih a personnal program.
Then you have to make an API request which consists in sending your HTTP request (using for example the post() function) to the API server. Your request will have to be set up beforehand with the right data according to the documentation.
The server will process your request, establish connections to the database(s), execute the request and bring back the data you requested.
The request will often be formatted in JSON, which will then have to be decoded.

 

Next ❯ ❮ Previous