We’ve developed some resources to help you work effectively from home during COVID-19 Click to learn more

Webhooks that does not trigger, a bug?

Hi,

We are implementing a webhook solution and I have some questions.

We are mainly at this stage focusing on webhooks for contact and person.

We have defined the following webhooks:

contact.created (Fire and works ok)
contact.changed (Fire and works ok)
contact.deleted (does not fire, when does this fire?)

person.created (Fire and works ok)
person.changed (fire and works ok)
person.deleted (does not fire, when does this fire?)

We sure want the abilty to add support for:
service.extratable.<TableName> .created/changed/deleted as well but not possible yet if I have understand correctly.

So my question is one does contact.delteted and person.deleted fire? 

 

We have discovered that when a contact or person is deleted from the system in the GUI a:
contact.changed / person.changed fires with som extra payload ( deleteddate).
That is logical since the contact/person is softdeleted. (ends up in the recylcebin)

What I think maybe is a bug, is that no contact.deleted / person.deleted is fired when the recyclebin is cleaned up by the system when background serivce does the hard delete. (eg removing the entry from the database)

If that had occured it would have solved our problem I think.

Another solution would have been to add a special event for the soft delete part.

contact.softdeleted / person.softdeleted and maybe change the contact.deleted / person.deleted to:
contact.harddeleted / person.harddeleted (fired when cleanup process run)

what I also found out when testing was that performing a restore from recyclebin a contact.chagend / person.changed is fired. That is logical since the delteddate is removed (making the entry visible in the GUI again)

Can someone share some thought about when contact.deleted / person.deleted webhooks are fired ?

Tests where performed in the 8.5 R04 version of SuperOffice.

best regards

Anders

 

 

 

 

 

 

RE: Webhooks that does not trigger, a bug?

Hi,

We need an answer :-)

Does anyone know anything about above described issue ?

/Anders

Av: Anders Larsson 16. jan 2020

RE: Webhooks that does not trigger, a bug?

Soft delete vs hard delete: https://community.superoffice.com/en/developer/blog/soft-delete-%E2%80%93-the-undo-revolution/

are the ones you are waiting for actually still in the recycle bin.

Do you have more info in a ticket so it's easier to look into it? Like, is this for Online or on site? If on site, how is this set up?
Since this is done by the ejCron: Webhooks should be configured true on the background job that cleans up, if it is running separately from the main site.

Av: Margrethe Romnes 16. jan 2020

RE: Webhooks that does not trigger, a bug?

Thanks Margrethe!

I have read that article a couple of times. 

The background is that we work on a solution that will send messages to a tibco messagebus, that other systems of the customer will then listen to. Then it is lite bit problematic if we cant catch when the deep-delete happens.

But if I understand correctly the deep delete (scheduled function that cleans up the recycle bin) should fire contact.deleted and person.deleted ? But I might need to add/alter some switch regarding ejCronSchedular ?

We run in on premise environment, at this momen 8.5. R04.

As for now we can catch the contact.changed, person.changed with the extra payload deletedDate to find out when a company/person enters the recylebin. But not when it is cleaned (deepdeleted). We can also catch when a company is restored from the recylebin. (ordinary contact/person.change , would be nice with extra payload there as well deletedDate:reset etc)

As other systems will take action on events generated in superoffice we would not like to send a delete event to the tibco messagebus until deep deleted has happend. Therefore I hoped for contact.deleted /person.deleted should be fired when the cleanup process runs.

at least this article states that there should be webhooks events of type contact/person deleted.

If not fired by the recycle bin clean up process, when are they fired? or are they obsolete?

https://community.superoffice.com/en/content/content/netserver-sdk/netserver-8.x/superoffice-webhooks/

Best regards

Anders

Av: Anders Larsson 16. jan 2020

RE: Webhooks that does not trigger, a bug?

Hi Anders, one of my colleagues had the same issue the other day. We're also using it to sync with another system, and noticed that the person.deleted event didn't fire, and came to the conclusion that it would probably be triggered on the deep-delete. Not sure if we tested and verified though.

You could probably test it by changing the preference to do the deep-delete after only 1 day. Wait until the next day, and see if it did anything.

Av: Frode Lillerud 16. jan 2020

RE: Webhooks that does not trigger, a bug?

Hi Frode!

Yes, that is what this thread is about.

Does the system generate any webhooks event when cleaning up or do we have a bug there...

Yes I actually did that last week before I started this thread and nothing on the registered webhook row for contact.deleted / person.deleted. eg counter +1 hit. etc. When the records in recycle bin was deep deleted and gone from the database. that indicates to me that background cleanup does not generate thoose events. Right or wrong , I do not know. 

But in my world events should be fired from that process.

If you get this to work I am interested in what version you run and so on.
Lets follow up on this.

/Anders

Av: Anders Larsson 16. jan 2020

RE: Webhooks that does not trigger, a bug?

Margrethe, can you elaborate on "Since this is done by the ejCron: Webhooks should be configured true on the background job that cleans up, if it is running separately from the main site."?

Does that mean we should put something like this into ejcrontab.txt?

webhooks = true

 

Av: Frode Lillerud 17. jan 2020

RE: Webhooks that does not trigger, a bug?

Is Service and Sales using the same web services url?

So does the nsEndPoint in the Service config file: https://community.superoffice.com/en/technical/documentation/administration/config-ini/serviceconfig.config/

have Webhooks = true in the web.config?

<Webhooks>
<add key="EnableWebhooks" value="true" />
<add key="RequireHttps" value="true" />
<add key="ValidateHttps" value="true" />
</Webhooks>

Av: Margrethe Romnes 17. jan 2020

RE: Webhooks that does not trigger, a bug?

Hi, Thanks.

Will check the settings. 

Probably service in our case uses the internal endpoint. (Bit same installation).

But Will dig into this as soon as possible, and mer you know.

Av: Anders Larsson 17. jan 2020

RE: Webhooks that does not trigger, a bug?

Hi,

I have now investigated further.

Setup:

SuperOffice Web (ak /sales) are setup using http local adress.
But with values:

<add key="urlPortOverride" value="443" />
<add key="urlSchemeOverride" value="https" />

Since we are using proxy/load balancing over Big IP F5 appliance

Service configuration use the local netServer adress (http) in the above sales (web implementation). As it is still not possible to use https in config for service...

In web.config for /sales /ak also the used netServer we have default:

<Webhooks>
<add key="EnableWebhooks" value="true" />
<add key="RequireHttps" value="true" />
<add key="ValidateHttps" value="true" />
</Webhooks>

Thinking of making RequireHttps and validate https = false to see if that makes any difference ?

the ejCronTab.txt contains:

REM These lines are executed by ejSchedule once every minute
"D:\Applications\SuperOffice\SuperOffice_CustomerService\bin\importMail.exe" tstcrm.xxxxx.intra.local
"D:\Applications\SuperOffice\SuperOffice_CustomerService\bin\ejournalCron.exe" tstcrm.xxxxx.intra.local

The deep clean up / emptying the recyle bin works. ( had 1 company and 1 contact) both deleted but no hit of webhooks registered. (contact.deleted or person.deleted) that if I understand correctly should be registered in this process? 

Other webhooks generated from the GUI and via REST-api works as expected ( eg person/contact.created and person/contact.changed)

select * from crm8.WEBHOOK
shows neither value in total_calls or total_errors for person.deleted/ contact.deleted.
state = 1, so reciving url are correct and working.

in crm8.registry table I have set reg id=186 to: Force use of https protocol  = 1

To solve probelm with buttons etc using SCIL ( for example edit button in marketing view ) 
(http not translated to https when using above described setup using http behind proxy)

/Anders

 

 

Av: Anders Larsson 20. jan 2020

RE: Webhooks that does not trigger, a bug?

Hi,

Have changed from 

<Webhooks>
<add key="EnableWebhooks" value="true" />
<add key="RequireHttps" value="true" />
<add key="ValidateHttps" value="true" />
</Webhooks>

To:

<Webhooks>
<add key="EnableWebhooks" value="true" />
<add key="RequireHttps" value="false" />
<add key="ValidateHttps" value="false" />
</Webhooks>

To see if that leads to any effect.

@Frode => Have you been able to validae if you get contact.deleted or person.deleted when ejournalCron.exe has run and hard deleted  / emptied the recycle bin ? 

@Margrethe/RND => In a perfect world and everything is correctly configured everywhere , ejournalCron.exe operations should trigger a contact.deleted , person.deleted when recycle bin is cleaned ?

If not i think it is a bug and that cleanup should result in contact.deleted and person.deleted.

For testing purposes: Is it possible to force cleanup by manupilation of deleted date on a contact/person and then run ejournalCron.exe manually ? 

 

Best regards

/Anders

 

 

 

Av: Anders Larsson 21. jan 2020

RE: Webhooks that does not trigger, a bug?

Hi

Changing web.config for netserver used by ejcron did not lead to any webhooks for 

contact.deleted / person.deleted ( comapany and person was removed/hard deleted from the recycle bin / database) 

To:

<Webhooks>
<add key="EnableWebhooks" value="true" />
<add key="RequireHttps" value="false" />
<add key="ValidateHttps" value="false" />
</Webhooks>

Any other suggestions? 

Is it supposed to be generated delete events when recycle bin are emptied ??

/Anders

Av: Anders Larsson 22. jan 2020

RE: Webhooks that does not trigger, a bug?

Hi Anders.

Marek will provide a more in-dept answer, but short answer is unfortunally no it will not trigger the delete event as you have correctly verified.

Av: Margrethe Romnes 22. jan 2020

RE: Webhooks that does not trigger, a bug?

Hi,

Thanks Margrethe, 

Looking forward to Markes deeper "longer" answer.

I guess this then is by design? Or is it a bug at the moment? 

Where would contact.deleted / person.deleted webhook events get created if not a bug?
Never? 

/Anders

Av: Anders Larsson 22. jan 2020

RE: Webhooks that does not trigger, a bug?

Hi,

When will Marek, be able to present a more indepth answer regarding contact.deleted / person.deleted.

What i need to know is if those events are obsolete or not. Accoring to this article it should be fired. but when? 

https://community.superoffice.com/sv/content/content/netserver-sdk/netserver-8.x/superoffice-webhooks/

Best regards
/Anders

 

Av: Anders Larsson 27. jan 2020

RE: Webhooks that does not trigger, a bug?

HI Anders, 

The reason why these webhooks events are not fired has to deal with where the cleanup actually occurs.

Webhook events are only performed when executed in the context of NetServer, and are the consequence of SoftTriggers.

The recyclebin cleanup of contacts and persons are processed outside NetServers normal data access routines, and therefore no .deleted events raised.

Functionally, this is not a bug, nor is it by design. Therefore, some research will have to be done to see how we can improve the current state of this and ensure deleted events are raised as expected.

Best regards.

Av: Tony Yates 27. jan 2020

RE: Webhooks that does not trigger, a bug?

Hi.

@Tony => Thanks for answering and clearifying a bit.

Yes, please do some research and implement, and let us know when we can expect it to be released.

From my point of view i would have expected contact.deleted and person.deleted when backgroud "hard" delete is performed.

Thinking out loud:

Maybe it would / should be possible to have contact.softdeleted, person.softdeleted ( when they are put into recycle bin) and contact.harddeleted person.harddeleted  when they are removed from the recycle bin.

We have an idea of hoe to "build/code" around this issue in our solution since we parse all the events through our "eventmanager" and creates our own event(s) to be sent of to the the customer messagebus api.

Best regards
Anders

Av: Anders Larsson 28. jan 2020

RE: Webhooks that does not trigger, a bug?

Hello,

Possible workaround for this would be to monitor the recycle bin using CRMScript and send an HTTP call yourself to your external "messagebus".

Something like this:

- Daily Script that looks up all persons/contacts with a deleted date > 1760-01-01, Put the id's in an extra table

- Daily Script that checks of the id's are still in the table (contact, person),

  • if gone it means they are deleted, send HTTP call
  • if still there but deletedDate is null or 1760-01-01 again it means they are undeleted, remove from extra table.
Av: David Hollegien 28. jan 2020

RE: Webhooks that does not trigger, a bug?

Hi there...

The design of the soft-delete system is as follows:

  1. Inside NetServer, everyone is still issuing DELETE "sql" commands
  2. NetServer database layer logic detects that you're trying to delete a row from a table that has softdelete functionality, and (at the last second) changes the DELETE into an UPDATE
  3. The relevant update hooks fire

At some later time, the cleanup runs. The cleanup code is part of netserver, but below the usual database layer. It does the following:

  1. Delete soft-deleted records that are over their cleanup date, which is the deleteddate+[number of days in recycle bin]
  2. Cleanup all dangling relations for that root entity (person or contact)

Cleaning up relations means that records that exist only as extensions of the root (phone numbers, email addresses, udef) are deleted; while records that have their own reason to exist (appointment, sale) have the relevant foreign key field (think person_id or contact_id) set to 0.

This is quite a deep operation that touches somewhere between 20 and 30 tables, and it's partly coded, partly dynamic. An important property is that it's not specifically cleaning up after whatever person or contact got hard-deleted: it cleans all records that point to nonexistent contact and person roots, and it neither knows nor cares why the root record is gone.

The reason this happens below the usual NetServer is partly that we need this to be available from the database-setup level; and also because it makes it possible to optimize the SQL beyond what NetServer allows us to.

It's not very hard to introduce a webhook callback possibility into the code. The question is more, what events do we want to trigger? Should we pretend that a soft-delete is a real delete, and what happens if someone picks the contact back out of the recycle bin? Should we differentiate between a delete on some other table, and hard-delete on contact?

There are at least three different ways of doing this:

  1. contact.softdelete - contact.delete
  2. contact.delete - contact.harddelete
  3. contact.softdelete - contact.harddelete
  4. contact.<whateverkindofdelete> - contact.undelete

Second, what about the relation cleanup cascade? We have salespeople who have accumulated many thousand records in their diaries. Delete one of them and you'd get a real flood update and delete webhook calls... do you want them, or is it better to let relations die in peace?

Viewpoints welcome!

Av: Marek Vokáč 28. jan 2020

RE: Webhooks that does not trigger, a bug?

Hi!

Thanks for answering Marek!

Thanks for possible workaround suggestion David!

 

So taking this forward i guess is a bit difficult and need some thinking. Now when we do know a little bit more about how the cleanup process works and what parts are involved we can discuss further.

From my point of view:

1. Yes I think the cleanup process should raise events when performing the hard delete of persons and companys.

as it is now we can not monitor by only webhooks to know when a contact/person i gone from the database.

Also as it is today deletion and restore of data from the recylce bin produces the same event. (eg contact/person.change + deletedDate as extra payload. I think it is impossible to say just looking at the payload to sayt if it is a delete o a restore that has occured.) 

2. I think it would be great with more granular events raised.
- contact/person.softdelete (eg we know that the entities are sent to the recylce bin) ( instead of changed with deletedDate )

- contact/person.harddelete (or just deleted) ( at the moment when cleanup process is finished with actually hard delete, record is gone)

- contact/person.restored ( this would be raised when soft-delete are undone by user o code, eg restored from recyle bin)

Also (maybe optional as setting or parameter for a webhook) would be to add data to the webhook-event.(payload)

eg to be able to "see" values of changed data. eg deletedDate, Number-fields etc

3. As stated before in the forum(another thread,but linked to this case)
I should also like to have webhooks enbled for services extra tables.
eg when adding, updating or deleteing records in such table.

 

Marek, your last comment:

Second, what about the relation cleanup cascade? We have salespeople who have accumulated many thousand records in their diaries. Delete one of them and you'd get a real flood update and delete webhook calls... do you want them, or is it better to let relations die in peace? 

Are you here talking about events on the activity entitiy ? 

Or does is it linked to when deleteting contact/person 

Best regards

Anders

Av: Anders Larsson 29. jan 2020

RE: Webhooks that does not trigger, a bug?

Hi Marek,

 

wouldn't it be usefull to divide all deletes in an action triggered by a user or triggered by the system?

Manually delete a company from SuperOffice (soft delete): contact.userdelete

Delete company from recycle bin (hard): contact.systemdelete

Manually delete a company interest: cinterest.userdelete

Cascade delete a companyinterest because company is deleted: cinterest.systemdelete

That way you can tap into the user triggers, system triggers or both (most of the time I suspect you would need the user triggers).

Regards,

Rob

Av: Rob Jansen 29. jan 2020

RE: Webhooks that does not trigger, a bug?

Hi Rob!

I quite like what you are sugesting here, dividing the deletes per user/system.

Lets see what Marek and others has to say about this.

From my point of view the important thing is to know when a company/contact really is gone from the system, stil active in system or are on the way out (eg in the recycle bin).

should be easy to understand just by analyzing the payload of the event raised. (eg Payload.event,) 

 

/Anders

Av: Anders Larsson 29. jan 2020

RE: Webhooks that does not trigger, a bug?

So instead of just contact.delete, you'd have contact.softdelete and contact.harddelete?

Could make sense. Sometimes I'm interested in one of them, and sometimes the other. Depends on what type of integration I'm making.

Av: Frode Lillerud 29. jan 2020

RE: Webhooks that does not trigger, a bug?

Hi,

RND => Let us know when you have considered our thougths and keep the dicussion alive :-)
Also let us know in advance what you plan to add/change around the webhooks event generated (especiall when the daily cleanup process runs so that the webhook flow will be complete ,eg covering "hard"-delete of contact and person records)

If needed/wanted i will galdly have Teams meeting etc to discuss possible solution further?
Anyone eles ? Frode?  

 

Best regards

/Anders

Av: Anders Larsson 4. feb 2020

RE: Webhooks that does not trigger, a bug?

Hi,

I am interesed in progeress of the path for this with webhooks generated by clean up process.

So, @Marek has RND come up with some clever solution ?

I can participate in discussion if you like and also show what we have built for a customer so that you get the whole picture of what we need to get a complete and logical solution for sending events of to an external message buss.

Best
/Anders

Av: Anders Larsson 20. feb 2020

RE: Webhooks that does not trigger, a bug?

Hi,

Has RND made any decision in any direction regarding webhook events as discussed in this thread??

Any new regarding REST-API in generation 9 of SuperOffice?

 

Best regards

/anders

Av: Anders Larsson 11. mai 2020

RE: Webhooks that does not trigger, a bug?

Hi Anders,

Yes. We have made some progress on this front, but only in terms of dialog and how and where to implement it. I do not believe any actual work has been performed to solve this yet. Release: TBD.

Best regards.

Av: Tony Yates 11. mai 2020

RE: Webhooks that does not trigger, a bug?

Ok, Thanks for the update Tony.

/Anders

Av: Anders Larsson 12. mai 2020

RE: Webhooks that does not trigger, a bug?

Just commenting this a bit as we also need contact.deleted/person.deleted webhooks for integration. I would have assumed deleted-webhooks fire when recycle bin is cleared. As not all systems have recycle bins so that's more universal logic. But of course adding more granularity (hard/soft deletes) isn't bad either.

Just hoping deleted-hooks get implemented soon enough :-)

Av: Mikko Tillikainen 19. mai 2020