Recycle Bin - what does the actual deletion?

Aha, ok. So for the onsite customers running 8.3 R01, where Recycle Bin is available already, the contacts won't be deleted until the next time they get upgraded to a new version, correct?

+1 Would be interesting to know

Correct. 

Ok. Even though Online customers have a tight and predictable upgrade regime, let's not forget that OnSite customers do not. It is common for OnSite customers to go 1-2 years between upgrades. We still have lots of of customers back on 7.5. So preferrably there won't be too many of these half-completed features in in the future, I hope.

Hello,

To add to the comment of Frode, it would be great if things like this would be added to the release notes...

With the GDPR in effect this really is a bad experience for OnSite customers, it effectively means that contact and persons that you delete are never deleted from the system (until you upgrade to > 8.3 R02 which could be > 6 months later).

Luckily we have only upgraded our own installation to 8.3 R01, we will wait until R02 before upgrading any OnSite customer.

The next release will delete companies (table contact) as well.

One reason we're releasing this "one step at a time" is that, as we consider it, it's no worse than before. In the past, if you deleted a person you would (from NetServer) just delete the person record and leave everything else dangling. With 8.3 R01, you actually see the same thing as long as you work through NetServer or COM, because all queries automatically get a WHERE addition that excludes soft-deleted rows. So even if you see them in the Recycle Bin, other parts of the system do not and you essentially get the same behaviour as before.

Then, when the deep-delete job comes along, it hard-deletes the overdue soft-deleted person records, and then it clears up broken relations. That part is not person-oriented ("let me clean up stuff left by person_id X"), it's the same logic that did the big consistency cleanup a couple of releases ago. Anyemail record that has person_id != AND person_id NOT IN (select person_id from person) gets deleted. When we release the contact deleter, it will be the same principle - follow broken relations and either delete records or set foreign keys to 0, depending on the table.

Thus, releasing soft-delete in one iteration and hard-delete later on, will result in much the same experience: until you upgrade to the version that has hard-delete, relations won't be cleaned, just like in the good old days. The difference in the meantime is that you actually get to see the persons in your Recycle Bin (and can Undelete), instead of them being gone but leaving a mess behind. It was also a requirement to be able to release Bulk Delete; we can't afford to spend the time deep-deleting during a bulk operation, so we needed soft-delete for that.

I appreciate that upgrading OnSite often is not feasible, which is why there is now a longer interval between designated OnSite versions. And we'll try to make sure that such versions contain consistent feature sets. I think the stagewise release of soft/hard delete is not ideal, but we did consider it and think it is not that bad.

By setting the preference to -1 you force deep-delete of everything that is in soft-delete status, regardless of age.

Hello Frode,

8.3 R02 only automatically delete's persons in the recycle bin, companies is next release (see release notes r04)

The ejournalcron does the deletion, we used the -force flag to force the deletion (since we had persons that where already deleted for some time in the recycle bin). Not sure what the timing is for deletion when running from the ejScheduler.

Do you have some more information about how the deletion works? I've been running 8.3 R02 a little while now, and the preference is set to only 1 day. Still my Recycle Bin is full of companies and persons that I deleted weeks ago.

Does ejScheduler perform the check every minute? Or once per night?

Hi Marek, I get what you're saying about the companies and persons being soft deleted, and thus not available via API methods. Reality is, unfortunately, not as black and white as that, but more "spotted grey". Our most GDPR-paranoid customers on 8.3 R01 is not going to be delighted about the fact that all the persons they cleaned out before GDPR took effect, are still visible for anyone via the Recycle Bin, and will be so for months to come.

Even when using the API's these softdeleted entities will pop up wherever we've used the SearchEngine method 'bypassNetServer'. There are very valid reasons for setting it to true, for instance where we do stuff like SUM's, grouping etc.

Also SQL is used heavily by some third party modules, like Business Analyze and Adwizas products. I also noticed that deleting a company no longer creates a deletion-record in TravelTransactionLog, but an update-record instead. That might be relevant for some solution down the road, dunno, but it's nice to know.

Point being, we need to know what we're dealing with here, so that we can take additional steps on our side as well. That's why it's very interresting for us to know how all this new soft-delete stuff works.

In 8.3 R02 SuperOffice thankfully deletes the persons. My _guess_ is that it is done once every night, but it would be nice to get some hard facts about this.

When it comes to 8.3 R01 (persons and companies) and 8.3 R02 (companies), I'd like to make a workaround for deleting the records that are overdue for deletion. Question: If we make a script that removes the rows from crm8.contact where DeletedDate is overdue, will the cleanup mechanism in the upcoming version remove all broken relations and references later?

Just to make it completely clear: The business logic in NetServer, Service and Win / COM API all automatically exclude rows that have a DeletedDate != null, so for all practical and functional purposes the records are gone. They will not appear in searches, selections, mailings, ... users of the system will see exactly the same behaviour as before, and so will integrations, except those that access the database directly.

Once the permanent, deep delete kicks in, it will clean up all records that are due for deleting. So if you have accumulated lots of soft deletes over, say, several months, all of them will get cleaned up once you install a version that has the permanent delete.

Does this mean that when you delete a company in 8.3 R02, the system only sets DeletedDate on the row in CONTACT-table? YES. 

Soft delete only adds a flag (DeletedDate) to the corresponding record, no move of records is done. And the business logic knows it should exclude the soft deleted records form the GUI.

The companies in 8.3 R02 will be soft-deleted = hidded from GUI but still kept in the DB. They need to wait for the next Onsite release to be able to get rid of them from the DB. 

As Marek wrote, hard delete not only deletes the rows but also clean ups the broken relations and references, so it is the best to leave the job to the process and do not try to delete them manually.

Hello,

Does this mean that when you delete a company in 8.3 R02, the system only sets DeletedDate on the row in CONTACT-table? So it hides the visibility but it still exist 'in the same way' in the database (?). Or is the row/information moved to a different table? 

Is it possible to disable this feature?
I'm thinking of the customers that just upgraded to 8.3 R02 and are going through their database to remove all companies they no longer are allowed to have any information on. If the row is not moved/deleted then how can they clean things up? 

//Eivind

Hi

I am hijacking this thread.

does "Hard/deap delete" results in a webhook event (contact.deleted , person.deleted ) ?

See a more deep explanation of my situation in this thread:

https://community.superoffice.com/sv/developer/forum/rooms/topic/netserver-api-group/web-services/webhooks-that-does-not-trigger-a-bug/

we are down the road now and we have also found that webhook event contact/person.updated is generated when a contact/person is deleted...

Best regards

Anders