Better description - rephasing the problem now in new section donw

Question: how to organise ./manage migrate on client in following setup:

I have to create django web site on client machine, with local usesr to login there and do some stuff locally.
There is also server with huge django application, which have its users doing things on the server.
And there is "just python" application on client machine, which gets data transtferred from server, stored locally and work with the local copy with direct SQL queries (lot of server table-connections are not used there), modifiing only two local tables, which structure (but not data) is copied from the server for consistency.

The server django application and the client "just python" applications have already established protocol for connecting and transfer data, purely from server to client. Not all tables are transferred (as not all are needed), sometimes only a structure of table is enforced on the client, sometimes also data.

The tables are transferred just as content of "normal Mysql tables" via some XML/ssh/whatever, losing its django connections, mainly no auth.* / admin.* / session.* / content.* tables are transferred (as irelevant), while a lot of django model have "modified_by_user" ForeignKey to (nontransfered) User table. This does not matter more on client.

No I have to create local django web on the client, with - its own set of Users, some profiles (OnToOne connected to the local users) - its own local tables which are read/write for local uses, but do not collide with the tables from server - shared tables (copied from server, purely read-only on client, overwriten sometimes by the sharing protocol)

I am writing the client side django part from scratch, I have acces to source of the server part (so I can copy models.py for example), but I need to add

local authentizations
manage content of "local only"tables
be able to expand them by migration, but do not change by migrate the existing "from server copied Mysql tables"
use some data for shown only from such "server copy tables" (some would be linked by "ForeignKey" from those local tables.

There is hard promise, that those copied tables would not do anything wild (will just grow in length and existing records would at maximum have some BoolienFields modified, maybe later there would be added more fields, but none deleted or changed - either by structure or by meaning))

Question is how I should construct my "client django application" to be maintainable, coexistet (and work with) the server copies of tables.

I can create my app with declaring some models, migrate, copy the "server models" migrate with --fake this part and then continue as usual, but it would be installed on more clients, and the structure of the shared tables may later add something, that I would need to read - so another --fake migration

Is there a way, to make some migrations "--fake" by default, so it would not really migrate at all, but would not be offered as unfullfilled (if those models would change)? Or I just have to name all of such migrations "0007_PLEASE_FAKE_ME' and do it by hands each time on each client? Or does it need totally different access?

There is not data sending from client to server and serve can send any data/structure changes it wants any time down. It should not make problem to my application anyway and if some structural changes should be reflected, I would be warned before and it could be done days/weeks after the transmission really happened.

I even can send my requests for model changes to the "server" staff to incorporate it in their models.py and push it down as "structure-only update"

I can incorporate their models.py to mine also, if it changes.

Still I am not sure, how my OneToOne profile to User would work on server, where is a lot of already established Users and none Profile model ...

the Djago version is the same on server and client (1.10), all tables are MyISAM.

Thanks for all suggestions

Edits (for @Thushar):

Clients usually run 24/7/365, tables are transferred in logical order by dependencies, first those, whitch end leves, then those, which reference them, core referincing all others last - so the the database model is everytime consistent (just maybe not complete in transfer, anything used have there prequisities already transffered)

'Created_by' is not used (de-refferenced) on clients. So far the clients run just some python code, which uses enumerate subset of colluns a tbles. this works for like 5+ years.

Modifications goes first to server (which fills values to tables), than tables are transferred to clients, then clients get SW update, which uses the new features (so it works each time by old way or by new way with complete new data) Server uses django, clients use plain SQL select name,filename, volume from themes where active=1
database scheme changes are done first on server, then transferred to client, then client sw is update to use the new collumns (like SQL ... AND plaing_allowed=1. Changes are back compatible, so even client with 10 versions obsolete SW is able work totally actual data, just missiing some new features.
the API for database transfer is pretty stable (even if little slow), transfers some form of TXT/XML data over SSH (many times no other connection is available to client) - with table descriptions first, then each changed line in order - and the order of transfers is structured so, that could be interrupted at any moment without affecting client ability to work. SW on client is updated after database, so there is (old client, old data - works) (old client, new data - works as previous version together)(new client, new data - works as new version)

old client simple ignores (does not know about) new tables/fields and is missing some new functions (also missing way to call such functions) so it works, like any changes was not doene at all.

About new recods (lines) - first are transferred dependencies (so ther is no way to use them), then more core tables(which then are sure, that all relations they uses are already in place) - the dependencies are tree-like, so it is possible (and done) order it ithis way, that leave are trasnferred first, nodes then, root last - (so if transfer is interruped, it is possible, that some leaves/subtrees are "stray"/unaccessible, but anything accessible have all dependencies in place) again, this already works 5+ years and tens or hundreds versions long. The client is so far more "device" than "computer" even if it runs full OS on PC platform - plans are made on server, clients just follows orders and do necessary computaitons, when needed.

Again the old way (django with mysql on server, python with mysql on clients) works reliabelly long years.

Not every time is client-server connection possible, then clients are suppose to work "locally" on old data, until the connection is available again (sometimes after hours, sometimes after months).

But "it just works".

My task is create new django application, which is to be realease on some (possibly all) clients, which adds some "local features" to existing schema, basically

have some new tables to place some plans and data into
have some web interface so users can log by web browser directly to client (which itself is little server with apache+django+mysql) and plan some local events there - those events are NOT backpropagated on main server and does not modify "official tables", just reads, what is avaiable and make "local notes"
those "local notes" then use already existing API on the client to notice it on more separate tasks, which needs be done "just now and here" - such API exist alredy and is used locally by some obsolete application (running on users PC/NB), which is to be replaced by web interface to that new django application on client.
the target is to be able expand the web interface to use some special "local tables" as well as read those "officially distributed" and expand the local API to allow more features, the local API for this (on client) just read data from tables and acts on them, not modifying anything in database. (New version of the local API should read on local tables too for more functionality)

My problem is, how to develope the client django application, with respect to that some database schema updates "just happens on background occasionally".

New version of the new django application can relly on that all new "background changes on global tables" are already in place, when it is released and only changes of local tables are needed.

The already existing system (optimalised for different criteria) is much larger, than the new part, so re-writing it from scratch is not option. Also there are some "unupdatable parts" such, that the old way of communication must be supported anyway in new version, as some computer tottally out of my reach depends on it. (But the communication protocol is designed to be expanded, so there is not big problem add some more informations there)

Better description

There is master computer M and identical slave computers S (1..x)
Master M runs Myslql + Django + Apache and have 1 API (M-S)
Master have tables TA (1..), TB (1..), TC (1..)
1. M-S API transfer table schema (CREATE TABLE like) for all TA and TB tables (either full or changes)
2. M-S then transfers data changes for TA tables
3. TC tables are not transferred at all
Slave S now runs Mysql + Old program and have 2 APIS
1. M-S API (for master over net)
2. S-local (XML based for old local communication by OldApp)
Master (not regullarry, but often enough, like 1-10x daily) contact each Slave and over M-S forces database updates on it
Slave does not update tables on Master (well there is some status reporting and such, but not mandatory and not interesting here)
Slave makes its own work (mainly playing music based on some hints from tables TA) independently (planning permutations ans such on its own)
when noted by XML from OldApp over S-local API the Slave stops what is doing and plays appropriate message, then continues
the S-local API is manipulated by some obsolete closed source application OldApp, which should be replaced by NewApp (see later)
- OldApp runs on users computers (totally different from Master and Slaves) under Windows and have hardcoded XML strings to send to Slave S
lots informations from Master M is irelevant for Slave S (mainly all authorisations and users, as decisions on slave are based on different criteria and TA tables are defacto read-only used here)
all of this works a lot of years and over lot of version changes an is too big to be rewritten in sane time.

New approach is to create and run Slave S based NewApp django application (+Apache) to offer better and new functionality, than the OldApp over web pages (hosted by Slave S)

To do so it neeeds: - authorisation (different from Master, bur all Master auth.* tables are in TC group and not transferred at all, so it can be locally reused on Slave) - - different for every Slave, not reported to Master - it reads some tables of TA group (but just read, does not modify content) - its own set of tables TD (1..x) to save - configurations and sounds and realtions - local to the particular Slave - - different for every Slave, not reported to Master - use S-local API (and expand it little - not a problem)

Questioned solution:

Copy models of TD tables on Master and let them to be transffered with TB tables to Slaves (just structure)
transfer auth.* tables on Master the same way (with TB group, only structures, not data) to Slaves
install new version on NewApp this way:
1. update TD on Master to synchronise with new version of TD in NewApp and migrate on Master
2. make transfer on Slaves (so new version is there forced)
3. update NewApp on Slaves and use just ./manage.py migrate --fake to stop complains

There are not problems with M-S API and its transfers (it just works, and default values in database are applied to new fields of existing records/rows), creating views and templates, (AFAIK if table for model contains more fields than model, such fields are just ignored)

This way Master would not use TD tables/model in any way (no interface, no admin, just being there empty) and Slave would be updated externally in sync with Master. Models.py on both would be the same in data part (maybe Master missing function of TD and slave missing methods on some TA/TB tables)

Questions

could it work this way (fake migrations)
are there some "system" tables needed (and automatically managed) by django on Master, which needs to be transferred (eventuall with data) to to Slaves (probably something about what migrations are done, or about structure of tables needed to meet models.py requirement) 2.1. is such tables are transferred, are even fake migrations on Slave needed? (IMHO there would be none, as all migrations are mentioned in those tables and the models fits)

from Migrations and two applications sharing parts od database on different machines

Hemant Vishwakarma

Sunday, 23 September 2018

Migrations and two applications sharing parts od database on different machines

Better description

1 comment: