Coming soon: Your brand new Help Center & Community! Get a sneak-peek here

How to catch CRUD operations performed by built in REST-API

Hi,

We have a customer that are building a solution ontop of the SuperOffice built in REST-api.

Is it possible somehow to "catch" when changes (CRUD operations) are happening to copmaines/contactapersons via the built in REST-API in SuperOffice?

If yes? How could it be done? 

We would like to catch this to be able to take action and for instance log this, or send notifiaction to other systems etc.

best regards

/Anders

RE: How to catch CRUD operations performed by built in REST-API

HI Anders,

Onsite installations can use:

  1. NetServer Service Events
  2. WebHooks

Online installations can use:

  1. WebHooks
  2. Zapier

Hope this helps!

Af: Tony Yates 25. sep 2019

RE: How to catch CRUD operations performed by built in REST-API

Thanks Tony!

That will help us taking the discussion further internally and also on the discussion with the customer.

Didn't tell but it is an on prem customer and based on that what would be the:

1. The most powerfull solution to use?

2. The most simple to implement but still catching when companies / persons are created changed

 

I guess this also anwsers some of my questions around logging in this thred:
https://community.superoffice.com/en/technical/Forum/rooms/topic/superoffice-product-group/crm-web-application/what-logging-are-possible-today-in-the-product/

I guess it would be possible to build a logging solution by using either of the above methods?
Are SuperOffice RND planning any bigger chnages around how and what could be logged?
I could bet that other customers also would like to have better oportunites to log things that happens in the application so I would like you to consider bulding a generic solution that would benfit all superoffice customers.

If you have time please comment in my thread about logging.

Best regards
/Anders

Af: Anders Larsson 25. sep 2019

RE: How to catch CRUD operations performed by built in REST-API

Just a thought about BulkUpdates.

I read in another post (don't remember the link) that BulkUpdates didn't generate Update Events when executed. I think the context was monitoring using WebHooks. Is this correct (I might have remember it wrong) and if so, is that still the case in current versions?

I suppose that NetServer Service Events are the solution that gives the best granularity when it comes to which events that are possible to monitor. As I understand it, using that kind of method you get access to all events available in NetServer, while WebHooks and triggers represent a wrapped subset of selected events to be able to monitor.

If that is true, will the NetServer Service Events-method give a possibility to monitor a lower level event that still makes it possible to monitor BulkUpdate-changes (if the above statement is true)?

Which level of transactional safety would be reasonable to expect by monitoring an event using Web Hooks vs NetServer Service Events?

I know that Christian Mogensen was quite clear in his ExpanderWorld-talk about Zapier, that the intent of that functionality was not for major integrations. I don't remember his exact words, but my interpretation was that he was telling the audience that, this is a pretty reliable notifier, but it isn't transactional. But Christan are welcome to clearify his current opinion on this, if he is reading this. :)

As you don't "own" and can verify the actual transaction, I would be reluctant to rate this kind of monitoring as a transactionally safe and "true" monitoring method.

In this specific case, I suppose that I would rely more on a NetServer Service Event-solution than a WebHook-solution, as you monitor nearer the source and are not exposed to potential network mishaps. Though, WebHooks have the advantage of being Online/Onsite compatible and might give more flexibility solution wise.

 

Best Regards

Marcus

Af: Marcus Svenningsson 26. sep 2019

RE: How to catch CRUD operations performed by built in REST-API

HI Anders,

My answers today always weigh our constant maneuvering towards online. With respect to compatibility, web hooks cover your use case and work in both on-site and on-line environments. They are well-defined events that will reliably send all subscribers notifications when events occur. The information that is send is minimal. It tells you what happened at the created|updated|deleted level, the identity of who was changed, and main fields pertaining to the event. It does not provide any entity data. You must use the identity key provided to look up any additional details you may require for additional processing/logging. To perform responsive api calls, you will need a means to manage credentials to perform subsequent actions.

With respect to NetServer Service Events (NSE)... They are extremly powerful in that you get the entire entity of interest, you never really know what event will be raised by webapi calls, or actions performed in the mobile/web application. That said, I published the Event Generator Utility on github to demonstrates how to create all NSE's, and show how to use Sysinternals DebugView application to monitor what events are being raised when a NetServer instance is invoked. Another benefit of NSE's is the events are raised in the content of the user, so it's trivial to know everything you may need to know about the use who raised the event.

There are other very low-level alternatives for on-site installations, but are not well documented, i.e. no tutorials or how-to guides. Maybe something for Expander World 2020?

  • Create a custom webhook plugin
    • receive notifications from pre-defined events and process in NetServer
  • Create a custom soft-trigger plugin (which the webhook implementation depends on)
    • subscribe to any table changes in the SuperOffice database, similar to database triggers.

None of these techniques are transactional. Transactional implies an exchange of messages between a sender and a receiver where each take turns to send or receive messages. They are the traditional publish/subscribe communication model. There is no opportunity for receiver to intrinsically influence the senders behavior.

If there is ever a chance this customer will migrate to the cloud, webhooks are the obvious choice.

Best regards.

Af: Tony Yates 26. sep 2019

RE: How to catch CRUD operations performed by built in REST-API

Webhooks are triggered by all the web service methods, so a AppointmentAgent.SetCompleted, AppointmentAgent.SaveAppointment, Appointment.Accept all trigger the Appointment webhook.

NetServer Service scripts are per method, so you would need many scripts to intercept all the different methods that can save or update an appointment:

  • AfterSetCompleted
  • AfterSaveAppointment
  • AfterAccept

Webhooks are definitely the way to go.

 

Af: Christian Mogensen 26. sep 2019

RE: How to catch CRUD operations performed by built in REST-API

...and what about the Bulk Update-feature? :)

Will the changes executed by the BulkUpdate trigger the expected WebHook-events or not?

/Marcus

Af: Marcus Svenningsson 26. sep 2019

RE: How to catch CRUD operations performed by built in REST-API

I think I might just be able to answer this, since I wrote the BulkUpdate engine...
It is based on the Entities objects like Entities.Contact in SoDatabase.
No service layer is involved so I suspect that means no events.

The service layer facade used by the web client is merely passing on a bulk update job description.

This is not going to change. Well not in my time anyway :-)

Note that a user requires a functional right to be able to use it so you can disable it for most/all users.

Conrad

Af: Conrad Weyns 26. sep 2019

RE: How to catch CRUD operations performed by built in REST-API

Entities and Row object changes are caught by the NestedPersist mechansim that underlies webhooks - so the bulk updates will fire webhooks.

Af: Christian Mogensen 27. sep 2019

RE: How to catch CRUD operations performed by built in REST-API

Hi Conrad,

Thanks for the clarification!

From a developer/integration perspective, I think it is important to make it very clear that possibly expected change-events will not get triggered when using the BulkUpdate, as it is important information for the ones that are thinking on building log/monitoring solutions.

If using the BulkUpdate via the API, you might be able to handle this separately using some workaround at least.

I have a case right now where we are looking on building an automated GDPR anonymization function where using the BulkUpdate via the API might be a more effective way of updating multiple records in a batch.

/Marcus

Af: Marcus Svenningsson 27. sep 2019

RE: How to catch CRUD operations performed by built in REST-API

Oh, Christian you were faster than me. :)

Ok, that's positive news!

If I'm not mistaken, the other post I read about, might actually have been related to BulkUpdate and the Zapier-connection. Is there any difference there or are the Zapier-app based on our Web Hooks and therefore should trigger the same way?

Might it have been differences in this workflow in an earlier version of the Zapier App, so a state valid at the forum post date might not be valid anymore? It could also be the case that the assumption in that post, was just plain wrong and the problem might have been connected to something else.


Best Regards
Marcus

Af: Marcus Svenningsson 27. sep 2019

RE: How to catch CRUD operations performed by built in REST-API

That's more related to how the Online system is configured. Batch tasks are run in a separate server, and that needs to be configured to enable webhooks. I suspect things there have been fixed.

That may have been the problem with respect to Zapier issues mentioned earlier.

Af: Christian Mogensen 27. sep 2019

RE: How to catch CRUD operations performed by built in REST-API

Hi!

Great discussion here :-)

@Tony, Thanks for claifying a bit about the alternatives

@Conrad/Christian , thanks for adding content/answers

@Marcus, thanks for adding more relevant questions / thoughts 

 

So to summerize :

1. It is possible to catch CRUD operations either by WebHooks or  NetServer Service Events

2. Method to use depends, but WebHooks seems to get the most votes since it can be migrated to/used in Online as well

As Tony mentioned and as an answer to that:
Yes, how to create webhook plugins/softrigger plugins would be a very valuble knowledge I belive. 

There are other very low-level alternatives for on-site installations, but are not well documented, i.e. no tutorials or how-to guides. Maybe something for Expander World 2020?

  • Create a custom webhook plugin
    • receive notifications from pre-defined events and process in NetServer
  • Create a custom soft-trigger plugin (which the webhook implementation depends on)
    • subscribe to any table changes in the SuperOffice database, similar to database triggers.

.

As I mentioned before it would be nice if SuperOffice Out of the box included a standard Log/Monitoring solution.

In a perfect world: 
It should be extendable and customizable (perhaps plugin-based then)  - Modular
It should be easy to understand and setup and maintain.

Extra features/plugins should be possible to add in an easy way, like a plugin manager, like in NotePad++ etc (that could perhaps connect to gallery of plugins, just like nuget/PS script gallery) 
It should be easy to test a plugin and then if not dessireable esay to delete

Plugin for notifing a message buss like TIBCO would be nice

Thoughts of above? 

Could be handled like a nice litle Open Source project that would benefit all partners/customer ?

Thanks!

Anders 

Af: Anders Larsson 27. sep 2019

RE: How to catch CRUD operations performed by built in REST-API

Hi!

@Marcus - Fun fact...BulkUpdate was created for GDPR. :)

@Anders - A standard Log/Monitoring system? Have you seen the diagnostics capabilities ? You could/should be able to use a standard debug viewer and watch the trace output. (link to github, cause SDK docs are muddled). Would something like application insights be better? How so...? What are the use cases?

A plugin system? You mean like the SuperOffice App Store? If not, what does the developer experience look like? We are talking about developers, right?

A soft-trigger plugin could certainly be used to send messages off to a service bus, even application insights.

I'm wary these days of creating such full-blown projects and open sourcing them. I see several people take our NON-PRODUCTION READY example code, shared to demonstrate barebones functionality, only to incorporating them into their applications, without any knowledge of the technology involved - or inclination to understand them, and use it as though its good to go! :( Then we end up in support cases where I see my code, same namespaces and no exception handling, etc...ugh! Rant over...

Production ready stuff is on nuget.org.

Best regards.

 

Af: Tony Yates 27. sep 2019

RE: How to catch CRUD operations performed by built in REST-API

Hi!

@Tony - Thanks again for answering in this thread.

What I ment by a standard Log/monitoring system, was that SuperOffice should include a framework/solution to setup more advanced logging/monitoring based on criterias that I choose. For example if I want to logg CRUD operations on Contact/contactperson changes this should be possible to setup. Or If want to take an action if such operation occurs like sending a message to a service bus or maybe send some data directly to an external system like Application Insights (will look into this).

With a plugin-system(mayb the wrong term) I ment that the above standard included framework/solution provided by SuperOffice should be extendable and possible for us partners to create addons, like if create a solution for logging/monitoring creation/editing of users in SO-Admin it should be possible to easily export/share this solution and import it into another instance of SuperOffice.

I am not a devloper, so forgive if I am using incorrect terms :-( However I am technical SuperOffice consultant that primarly work with big customers on prem (stil, yes). They sometimes have demands like logging/sending of data to a servicebus. 

I can see challanges with above, but I think that having the oportunity to set up logging/monitoring from with the SO Admin-client or a separate logging / monitoring  admin-client would benfint a lot of customer both in the onprem and online product.

How it should be done I can not answer, see this like a wish to be able to trace what happens in the system (in detail) and if dessireble log/monitor/take action and start flows etc.
Open Sourcing was more for addon-parts,eg that we as partners could benefit from sharing solutions that we build. 

Nether less I will look into: diagnositcs capabilites, debug viewer, application insights, soft-trigger plugins and nuget

I hope above clarifies what I am after.
Don't hessitate to take the discussion further or ask more questions, and I would gladly have skype session etc to discuss more details if needed.

Best regards

Anders

Af: Anders Larsson 1. okt 2019

RE: How to catch CRUD operations performed by built in REST-API

Hi,

Will hook into this thread with a follow up question :-)

We are now able to catch update and create (delete, read)  events on contact (company) and person objects. Via GUI, REST and netserver integrations. That is good.

What we now are wondering is if andif yes how we can catch events on service Extra Table.

We need to catch create, update and delete.

Is it possible by WebHooks?
or do we need to use another teqnique and then wrap it somehow as an own rest-api that delivers a custom web-hook etc that could then be catched and handled. Any suggestions?
Soft-triggers, as Tony has mentioned, seems to be possible way, since it can find changes on any table in the SO-database?

Cannot find somethin like: extra_table.created or Extra_Table.updated here:

https://community.superoffice.com/en/content/content/netserver-sdk/netserver-8.x/superoffice-webhooks

 

/Anders

 

Af: Anders Larsson 4. dec 2019

RE: How to catch CRUD operations performed by built in REST-API

Hi Anders,

No, we do not [yet] support webhooks on extra tables. We have discussed it and may introduce that option sometime in the [distant] future.

You can create soft triggers on extra tables. Softtriggers expect the table number, which is accessible via the Dictionary.

var dict = SoDatabase.GetCurrent().Dictionary;
var table = dict.GetFromTableName("y_equip");
var tableNumber = table.TableNumber;

In this thread, some references were made about not being able to receive webhooks notifications from Bulk Update operations. That has since been remedied. I go into more detail about it in another post

Best regards.

Af: Tony Yates 2. apr 2020