Orchestrating Puppeteer from Lambda with zero lines of code

Kumologica is a free low-code development tool to build serverless integrations. You can learn more from our medium channel or on our website.

In this article we will look into the details of how to use Kumologica Puppeteer node to capture website content.

Use Case

Having the ability to take screenshots of dynamic website content and storing it for later reference is a regular need in integration. Kumologica supports this use case by providing the Puppeteer node.

We will look into the steps of how to go about capture the results of amazon.com website search, and storing it in an S3 bucket for future reference and analysis.

Solution

We will develop a single flow that will be deployed as an AWS lambda function.

Outline of the use case implementation is as below:

• The flow will receive the URL that we need to create snapshot of in JSON format as part of the request.
• The flow will use Puppeteer node to take snapshot of the URL provided in PDF format.
• The flow will use AWS S3 node to save the PDF to an AWS S3 bucket.

Implementation

Follow the development process as outline below:

• Open Kumologica designer.
• On the popup Click Create New Project.
• Provide the Project Name and Location to save your project and click Save.
• Drop from the palette an EventListener node onto the canvas with the EventSource as Amazon API gateway , verb as POST and resource end-point as /snapshots

Figure 1: EventListener configuration

• Puppeteer node is part of contrib nodes. Click on the button “Add more nodes” to add the Puppeteer node to the main palette. The newly added node will be available under the category “AUTOMATION”.
• From the palette, add Puppeteer node onto the canvas. Configure it as outlined below.

Figure 2: Kumologica Puppeteer node configuration

• The result of this node processing will be available as nodejs Buffer object at: msg.payload.Dynamic-PDF.result
• From the palette, add AWS S3 node onto the canvas. We will use the AWS S3 node to store the PDF results onto an S3 bucket.
• Configure the AWS S3 bucket as outlined below.

Figure 3: Kumologica AWS S3 node configuration

NOTE: Content field of the S3 node supports dynamic JSONata based expressions. In JSONata, ‘-’ character has to be escaped with single quotes. If there is no ‘-’ character, it could be written with single quotes. eg: msg.payload.DynamicPDF.result

The resulting flow must be looking like this:

Figure 4: Finished flow

Testing

Let’s create our test case flow.

Click on the “Test” tab to switch to the test flow canvas.

Figure 5: Test canvas tab

• From the palette, add TestCase node to the canvas. TestCase node is available under the category TESTING.
• Configure the TestCase node as outlined below:

Figure 6: TestCase node configuration

• The TestCase node will target the message to the EventListener node. From the Target Node dropdown, select the EventListener node. We had named it “POST: /snapshots”.
• Event field will be populated with the sample event for EventListener type.
• Change the payload to the one below.

{
    "path": "/snapshots",
    "httpMethod": "POST",
    "headers": {},
    "body": 
    {
        "endPointURL": "www.amazon.com/s?k=digital+thermometer"
    }
}

• Path is set to /snapshots and httpMethod to POST. This is the resource endpoint configuration provided in the EventListener node.
• In the body, we will be passing the URL that needs to be used against the key endPointURL. During runtime, this value will be set into the message at: msg.payload.endPointURL
• From the palette, add TestCaseEnd node to the canvas. TestCaseEnd node is available under the category TESTING.

The resulting test flow should be looking like below:

Figure 7: Test case flow

• Select TEST tab from the right side panel.
• Select the AmazonSearchPDF from the drop down. This is the name that we gave to the TestCase node in the TEST tab.
• Click Run button. Verify the results in the Amazon S3 bucket.
• Download the PDF created.

Figure 8: Amazon search result captured as PDF document.

Conclusion

Thanks for reading (or watching the video tutorial). We hope you enjoyed this use case and appreciate how simple is to implement serverless integrations with Kumologica. We would love to hear your feedback, and don’t hesitate to contact us if you have any questions.

To learn more about Kumologica checkout our official documentation at kumologica.com.