Coming soon: Your brand new Help Center & Community! Get a sneak-peek here

NetServer 8.2 breaks Authenticate method for previous versions

So, previous versions of NetServer can not connect to an SO8.2 database. The Authenticate() method in previous versions look for the PHYSICALDATABASE table, which has been removed. I'm not sure if this break in compatibility was intentional or not, or if it's just an unintended side effect of the schema change.

Now, the *really* annoying part is that 8.2 has a number of database schema changes (we haven't yet investigated everything), so we can't use an 8.2 version of NetServer to connect to older versions of the SO database either (because it crashes when trying to fetch data from the tables with new columns).

I think I know what the answer will be, but still... is there any way that this could be sorted out so that we don't have to maintain two distinct release branches for every application that connects to SuperOffice using a local NetServer connection? This would presumably require adding the PHYSICALDATABASE table back into the database somehow - Is there any way we can do that in a way so that we can have the Authenticate method working again (since I'm assuming it needs more than just the physical presence of the table itself)?

We've always loved the fact that we didn't have to do this when it came to SuperOffice integrations - That an application written using NetServer DLLs from version 7 (and even SIX in many cases) would still work with SuperOffice 8 without any problems.

RE: NetServer 8.2 breaks Authenticate method for previous versions

We are also very concerned about having to maintain 2 sets of source code for each project we have written for SuperOffice. We have rebuilt several of our projects using 8.1 DLL's and these seem to work in both 8.0 and 8.2, is this a possible solution to this issue?

 

Av: Rich Hacker 5. feb 2018

RE: NetServer 8.2 breaks Authenticate method for previous versions

The breaking changes in 8.2 are there to give you more flexibility in creating & maintaining your own database tables.

Source code is essentially unchanged between 8.0 and 8.2 - NetServer authenticate works the same way as before, and the Services API is backwards compatible back to 7.5. If you have a solution for 8.0 NetServer, you should be able to retarget to the 8.2 NetServer and re-compile without any changes.

Not sure I understand the significance of the old PHYSICALDATABASE schema tables for authentication.

Av: Christian Mogensen 5. feb 2018

RE: NetServer 8.2 breaks Authenticate method for previous versions

I don't know why the Authenticate method in older versions of NetServer needs to check the PHYSICALDATABASE table either, but for whatever reason it does (according to the error message in the log). So an older version of NetServer will crash when attempting to authenticate with an 8.2 database, since that table is now gone. Adding a reference to NetServer 8.2 and connecting to an 8.0 database causes various crashes when retreiving data from tables that have new columns added to them.

Referencing a given NetServer version is well and good for an application that's installed on a single site, or where we're in a position of knowing which SO version every customer is running, but we're not in that position 90% of the time.

We develop and deliver standardised versions of our integrations, and we need to support SuperOffice versions at least all the way back to 7.5, and I believe we still have customers running 7.1 too. In many cases they are staying with old versions of SuperOffice because they have other integrations that will break if SO is upgraded, so in turn that means we still have to support these older versions too.

We'll have to do some further testing to see which versions of NetServer will now fail to authenticate and come back to this topic, but we're already seeing several of our applications crashing at customer sites once they upgrade to SO 8.2.

I will also have a look at Rich Hacker's suggestion and see if Netserver 8.1 will allow us to connect to both 8.2 and 7.5 databases.

I'll try to keep this thread up to date with whatever we find.

Av: Eskil Sæter 5. feb 2018

RE: NetServer 8.2 breaks Authenticate method for previous versions

Update #1:

I just tried adding a reference to NetServer 8.1 and accessing an 8.2 database, and that crashed.

Again, the error message seems to be pointing to the PHYSICALDATABASE table not being present:

Method not found: 'Boolean SuperOffice.CD.DSL.PhysicalDatabase.DatabaseManagement.IsCsExtraField(System.String)'.

at SuperOffice.Data.Dictionary.SoDictionary.LoadFromDatabaseModel(DatabaseModel databaseModel, DatabaseManagement dbm)
at SuperOffice.Data.Dictionary.SoDictionary.Load(SoConnection con) in C:\agent1\_work\34\s\Server\Source\SoDataBase\Data\Dictionary\SoDictionary.cs:line 330
at SuperOffice.Data.SoDatabase.Init(SoConnection con) in C:\agent1\_work\34\s\Server\Source\SoDataBase\Data\SoDatabase.cs:line 1423

Av: Eskil Sæter 5. feb 2018

RE: NetServer 8.2 breaks Authenticate method for previous versions

Update #2:

Tried referencing 8.2 NetServer and connecting to a 7.5 database. Authenticate method crashes. This time with a pretty clearly stated error message, so this is obviously not a bug :) 

Reading DatabaseModel from Pre80 not supported.

at SuperOffice.Data.Dictionary.SoDictionary.ReadDatabaseModelWithFallback(DatabaseManagement dbm) in C:\agent2\_work\13\s\Server\Source\SoDataBase\Data\Dictionary\SoDictionary.cs:line 560
at SuperOffice.Data.Dictionary.SoDictionary.Load(SoConnection con) in C:\agent2\_work\13\s\Server\Source\SoDataBase\Data\Dictionary\SoDictionary.cs:line 329
at SuperOffice.Data.SoDatabase.Init(SoConnection con) in C:\agent2\_work\13\s\Server\Source\SoDataBase\Data\SoDatabase.cs:line 1430

Av: Eskil Sæter 5. feb 2018

RE: NetServer 8.2 breaks Authenticate method for previous versions

Update #3:

Referencing NetServer 8.2 and contacting a 8.0 database. Authenticate works, but retrieving data fails.

Trying to retrieve a contact entity throws an error referencing a missing column:

Invalid column name 'CategoryFamily_id'.

at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
at System.Data.SqlClient.SqlDataReader.get_MetaData()
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
at System.Data.SqlClient.SqlCommand.ExecuteDbDataReader(CommandBehavior behavior)
at System.Data.Common.DbCommand.System.Data.IDbCommand.ExecuteReader()
at SuperOffice.Data.SoCommand.ExecuteReader(Nullable`1 behaviour, Int32 pageSize, Int32 pageWanted) in C:\agent2\_work\13\s\Server\Source\SoDataBase\Data\SoCommand.cs:line 509

Av: Eskil Sæter 5. feb 2018

RE: NetServer 8.2 breaks Authenticate method for previous versions

If I'm missing something obvious, then please point me in the right direction, but as far as I can tell right now:

  • NetServer 8.2 is incompatible with any database from an older version
  • Any previous version of NetServer is incompatible with an 8.2 database

Help?

Av: Eskil Sæter 5. feb 2018

RE: NetServer 8.2 breaks Authenticate method for previous versions

RE: Update #3:

We have the same issues when using 8.1+ DLLs connecting to an 8.0 Database, the error I found this morning was to do with the new column ''Family_Id" on the Category table missing in the old DB structure.

Also, If we use 8.0Dlls to conntect to an 8.1+ database with Custom Tables, we get errors when trying to access these tables, specifically: 'The Given Key was not found in the Dictionary'.

I am currently investigating these issues further and will give an update leter today.

Chjeers

Rich

Av: Rich Hacker 5. feb 2018

RE: NetServer 8.2 breaks Authenticate method for previous versions

Update #4:

Connecting to an 8.2 database using NetServer 8.1. But in this case, the PHYSICALDATABASE table is still present in the database (it wasn't removed during the upgrade).

Same error as before though. It still crashes with an error referencing PhysicalDatabase, even though the table is physically still in the database. If I had to guess, it might be an issue with the checksum value present in the table's row?

 

Method not found: 'Boolean SuperOffice.CD.DSL.PhysicalDatabase.DatabaseManagement.IsCsExtraField(System.String)'.

at SuperOffice.Data.Dictionary.SoDictionary.LoadFromDatabaseModel(DatabaseModel databaseModel, DatabaseManagement dbm)
at SuperOffice.Data.Dictionary.SoDictionary.Load(SoConnection con) in C:\agent1\_work\34\s\Server\Source\SoDataBase\Data\Dictionary\SoDictionary.cs:line 330
at SuperOffice.Data.SoDatabase.Init(SoConnection con) in C:\agent1\_work\34\s\Server\Source\SoDataBase\Data\SoDatabase.cs:line 1423

Av: Eskil Sæter 5. feb 2018

RE: NetServer 8.2 breaks Authenticate method for previous versions

None of the databases we're testing this with have any extra tables added, by the way.

Av: Eskil Sæter 5. feb 2018

RE: NetServer 8.2 breaks Authenticate method for previous versions

Hi Rich,

Category.family_id was introduced SuperOffice 8.1 and was then renamed in SuperOffice 8.2 to Cateogry.CategoryFamily_id.

That breaks compatibility of the 8.1 libraries for 8.2.

The 8.0 libraries are not compatible with 8.1, due to the changes for the SODictionary replacement (indeed, removal of physicaltable, conceptuatltable, etc).

The state of on-premises SuperOffice apps as we found right now is that you will need to drop exactly the right libraries for the current database otherwise it will not work. 

Av: Matthijs Wagemakers 5. feb 2018

RE: NetServer 8.2 breaks Authenticate method for previous versions

It's very simple:

The old 4.X - 7.X dictionary system was ripped out and replaced with a newer, more flexible one. There is no way that 7.x clients can talk to a 8.2 database.

Vice-versa: There is no way that 8.2 client can work with a 7.x database. The meta-data system is completely different.

Go look at the video from Expander World 2017:

https://youtu.be/Dd74Ki9IRic?t=8m0s

Av: Christian Mogensen 5. feb 2018

RE: NetServer 8.2 breaks Authenticate method for previous versions

Well that's certainly disappointing. We don't have a single application that touches the dictionary stuff, but still everything breaks if our customers upgrade to SO 8.2.

But alright; we'll advise our customers to avoid upgrading until we get our release branching and distribution sorted.

Av: Eskil Sæter 5. feb 2018

RE: NetServer 8.2 breaks Authenticate method for previous versions

Agreed Eskil, really suprised by this, expected to have to do work, to get all our products upgraded to 8.2, but was hoping it would be backwards compatible for our customer on v8 and below.  This is a real logistical nightmare in having to keep multiple versions of our bespoke maintained.

Av: Trevor Sharp 5. feb 2018

RE: NetServer 8.2 breaks Authenticate method for previous versions

Guys, a question. When you use NetServer in this case, have you configured it to talk directly to the SQL database, or do you use it in remote mode?

Gut feeling is that some of these issues could be avoided by using NetServer in Remote mode.

Av: Frode Lillerud 5. feb 2018

RE: NetServer 8.2 breaks Authenticate method for previous versions

Frode: I'm assuming you might be right about that, but switching to remote mode would require a pretty substantial rewrite though.

Unless I'm out of the loop (which is entirely possible), remote mode doesn't support things like entities, and we have to communicate through an entirely different set of methods?

Av: Eskil Sæter 6. feb 2018

RE: NetServer 8.2 breaks Authenticate method for previous versions

A solution to this is to upgrade your code to use the netserver 7.5 (since some cheese moved between netserver 7.0 - 7.5) and then use assemblyBinding redirects to point to the correct netserver version, without having recompile it.

 

Info: https://community.superoffice.com/documentation/SDK/SO.COM.IApplication/html/guideNetServerVersioning.htm

Av: David Hollegien 6. feb 2018

RE: NetServer 8.2 breaks Authenticate method for previous versions

David: Thanks, I'll give that a go and see if we can't get around this.


In the meantime, here's another update, and now it's getting weird.

Added references to NetServer 7.5 in my project and connected to an 8.2 database. Authenticate works perfectly on an UPGRADED database (where several non-8.2 tables are still present and weren't deleted by the upgrade), but crashes on a FRESH database (where these tables aren't present).

So NetServer 8.0 and 8.1 both crash when connecting to this upgraded 8.2 database, while NetServer 7.5 succeeds and seems to handle it completely fine, no issues at all. I can retrieve and update entities as normal (at least for the types I tested)

 

So just for fun(?) I copied these tables from a fresh 7.5 database into my fresh 8.2 database (schema and data):

- PHYSICALTABLE
- PHYSICALSCHEMA
- PHYSICALFIELD
- PHYSICALDATABASE
- INDEXFIELD
- DICTIONARY_BASE
- DICTIONARY
- DICINDEX
- CONCEPTUALTABLE
- CONCEPTUALFIELD
- CONCEPTUALDATABASE
- RELATIONSHIP

...and everything seems to be working fine using NetServer 7.5. Authenticate() succeeds, and everything seems to work like it should.

So I'm not sure what this accomplishes or proves, but I do find it very strange that NetServer 7.5 is so forgiving when it comes to database versions, while NetServer 8.0, 8.1 and 8.2 only seem to be able to communicate with their own respective databases and no others.

But I guess that in a pinch, it *might* be possible to manually add those missing tables to a new SuperOffice database and continue using your application. As long as it references NetServer 7.5 and not a newer version.

Av: Eskil Sæter 6. feb 2018

RE: NetServer 8.2 breaks Authenticate method for previous versions

Hi Eskil, in RemoteMode you have Agents and Entities, but not rows and OSQL.

So if you're using f.instance ContactAgent, then that'll work.

Frode

Av: Frode Lillerud 6. feb 2018

RE: NetServer 8.2 breaks Authenticate method for previous versions

Hi guys,

This is a complicated subject and I will do my best to explain. Get some fresh coffee and shutdown Outlook, this will take a while to read :-)

Like Christian Mogensen said, we've totally changed the way the database is described, i.e. how the database model metadata is formatted and stored. "Totally" means that instead of 9 tables that are read into a C++ data structure using custom code, we now have a database model defined in C#, serialized using JSON and then zipped to save space, in one record in one new table (databasemodel). You can reference the .CD.DSL and .CD.DSL.Implementation dll's in the LinqPad tool and explore it if you want to.

There were many reasons to make this drastic change - chief among them that the old system did not have space for new attributes and was very inflexible, as well as slow. With our transition to multiple teams and releases every 3 weeks we couldn't stay with a system where database changes happened once every 18 months, supported by one single person (me) who understood how the magic worked, and required a separate software installation per upgrade version (no 8.0 -> 8.3 direct upgrade).

This means that there is a fundamental compatibility break between 8.0 and 8.1. From 8.1 on, the entire series of steps leading up to a complete database resides in SuperOffice.CD.DSL.Database, including the priming data (say goodbye to sotables.ini and the IMP files). We can take a database that is at any 8.1+ stage and upgrade it to latest, using that single DLL, hopefully for years.

Up to 8.0, the dictionary tables conceptualtable, conceptualfield etc. contained the metadata. From 8.1 it’s the new databasemodel table. During upgrade from 8.0 to 8.1, we copy any partner-defined tables from the old dictionary to the new. And then we chicken out, instead of brutally deleting the old dictionary tables, we leave them there as fossils. I use the term fossil because they will no longer be updated, and as our database schema evolves they will become outdated – meaning «wrong». In a freshly created database you will not find them, we don’t create fossils.

NetServer does not contain too many compatibility checks with the database. The intention is that stuff should work for as long as possible - but the flip side is that incompatibilities between various post-8.1 releases appear as you hit them.

In a perfect world, that would be the end of it – one clean break between 8.0 and 8.1, and then a reasonable (maximized) amount of compatibility. However, agile development is not trivial to reconcile with the fact that databases are not thrown away with every release like the code is. You can experiment with code, but database changes are forever. Or almost: during the 8.2 development cycle we realized that we had to change the way we implemented Consent Management in 8.1, and we also had to change the database.

For 8.2 we therefore did something that we try to avoid: Change (as opposed to just add) published database fields, in particular the new field on the category table. As a result, we got another major compatibility break at the database level. Under the circumstances it was unavoidable, because living with an unsuitable database schema adds a lot to the lifetime cost of ownership. But it was painful. We will do our best to not do that again.

But, the reality remains: we are now *far more agile* than we used to be, releasing new code to Online every three weeks, and – at least in the first half of 2018 – with database changes almost every time. There is absolutely no way that we can guarantee that a netserver works with any other version of the database than what it was built for. Quite often it *will* work until you touch whatever functionality was added; but with the GDPR features we have to make quite extensive changes to existing, core entities such as person, and that will break things.

It will be possible to find specific cases that netserver version 8.x works with database 8.y; but these are not supported configurations. So one of the absolutely key takeaways is to *use the service layer* api’s whenever it is at all possible. Do not link directly to netserver if you can possibly avoid it. We maintain distinct versions in the service layer and guarantee compatibility there.

At the netserver and database level, 8.0 and 8.1+ simply are not compatible. Copying 7.x dictionary tables into an 8.2 database so you can run your netserver 7 version may work for a while, but you cannot and should not assume it does. You will also risk fouling up the database because your old netserver will do things the wrong way (functionally, but not crash), and you may discover the effects a good while later, when it’s really expensive to fix. Please don’t do that.

We’ll also come out with a statement on releases to on-site customers and partners (as opposed to online) – frequency, compatibility, etc., in the very near future. It has been an area of active discussion, and it is not an easy topic.

 

Marek

PS. The error message “Method not found: 'Boolean SuperOffice.CD.DSL.PhysicalDatabase.DatabaseManagement.IsCsExtraField(System.String)'.” does not refer to any table. It refers to a missing C# method and leads me to suspect that you are mixing dll’s from different netserver versions. That is not going to work.

Av: Marek Vokáč 28. feb 2018

RE: NetServer 8.2 breaks Authenticate method for previous versions

Thanks for the technical background on this Marek. I think the most important thing for us to take away from this is to use the service layer for all future development (we were already on this path the last years). We do have a lot of legacy that runs on NetServer core, so we have to deal with compatibility or rewrite it.

Perhaps it's also a good idea to put a big warning about compatibility in the documentation, for partners and customers that are currently not aware of this?

We are are also really looking forward to the statement regarding onsite releases.

Av: Matthijs Wagemakers 1. mar 2018