Alt Import method

[ippocratis]

Would like to browse to run the re mm ote server on my android device .
Can drag and drop obviously using an android device.
An upload button is needed in this case

[andrepxx]

You want to run what on your Android device?

The reason I use drag and drop is that web applications can only access files that the user "actively provides". Usually this happens by dragging them into the web browser.

The only alternative is an HTML file form element, which will open a system dialog that allows the user to select a file. However, the issue is that this UI element looks different on every platform and cannot be styled by CSS, so I really don't want to have it visible in the application.

There might be a workaround of using a hidden HTML file form element and triggering events on it via JavaScript, but I'd have to look into how easy / difficult this would be to implement in this context.

[ippocratis]

Running my services on a headless raspberry pi and access them over my phone.
I see that the button implementation is tricky.
A command line option then maybe?

[andrepxx]

This is already possible, effectively, by uploading the file via "curl" when the server is running. The only problem is that you need a session token and that's difficult to obtain without using the UI, since you have to complete a challenge-response based authentication to obtain it.

Script to upload would look like this.

#!/bin/bash

TOKEN="3lCV5aXwhuyKN3TVfix1Y2r29tiTjkVzQcwbtq6d6T443KUDyDoK5X97TUbuPmWDMRe20r865HXI46mKXzqT4A=="
FORMAT="json"
STRATEGY="newer"

curl --http1.1 -k -F "cgi=import-geodata" -F "token=$TOKEN" -F "format=$FORMAT" -F "strategy=$STRATEGY" -F "file=@/home/user/Downloads/Records.json" https://localhost:8443/cgi-bin/locviz

With "command line option" you mean an import as a "one-off operation" when the server is not running? (Like we have it for setting up user accounts, etc.) So you'd first create all your users, then do all your imports and then run the server?

[andrepxx]

So basically, we have two possibilities to implement this, if we actually want to integrate it into location-visualizer.

location-visualizer has "offline" operations, so to say, which can be performed from the command line when the actual server does not run. These are the ones for user account management (creating, deleting accounts, setting credentials, giving / revoking permissions, etc.).

EDIT: Well and importing and exporting tiles are also "offline operations".

And it has "online" operations, such as uploading geo data, which can be performed from the web interface when the actual server runs.

Now uploading geo data is already available as an "online" operation (meaning there has to be a running server already). But you can effectively only do it from within the web interface. You could do it via curl if you could get past the challenge-response authentication, but that's so far only implemented in the web interface.

Now what we could do is implement a command line option where you run locviz, provide a server address (which could be local or remote), credentials, and a file to upload (and the options about "how" you want to import it - you know, format, import strategy, etc.) and then it connects to the remote locviz instance that is the actual server, establishes a session, uploads a file, and terminates the session again. That would be possible. But then you run two locviz instances next to each other - one acting as a server and one acting as a client. It's also a change in scope of what locviz is. So far, it's only a server application, then it would also be a client for that server application.

I'm just brainstorming how to implement this with the least technical hassle and the least confusion to the user possible. I think providing a "CLI client" that can upload files is the right way to go (better than the "offline" / "one-off" option), since we already have the import as an "online" (= with server process running) operation and that CLI client would make use of it. But yeah, it is somewhat a change in scope. And it would not be that easy to script since two processes (server and client) would be running in parallel. But if you keep the server running all the time and then do a second SSH session and there run the client to perform the import, that would probably be the right way to do it.

Just brainstorming, so far. What are your thoughts on it?

[ippocratis]

The script method looks fine .
The problem is I don't know how to obtain my token.

Can't see any hint in the ui.

What I meant by command line option is similar to what you described with the curl script .

e.g. ./locviz import-geodata Records.json

[andrepxx]

Yes, I understand. But I'd like to go into detail once more about what I meant.

Currently, locviz can be run without options, which starts the server, or it can be run with options (like create-user ..., remove-user ..., set-password ..., import-tiles ..., export-tiles ..., etc.), where it perform a certain "one-off" operation.

However, these "one-off" operations shall only be performed when the server does not run. The reason is that they will, of course, run another process (independant of the server process - if there is one), do its thing, and shut down. So if the server were running while we also execute one of these commands, we'd have two processes (the server process and the one executing the "one-off" command) running in parallel, accessing the same database files (geographical database, user database, tile database) - each with their own file handles and "database drivers" - which is not what's supposed to happen and may lead to undefined behaviour, like the other process not getting notified of changes (as is the case for changes to user accounts, since these are only loaded once at startup), or even file corruption (which might happen, for example, when tiles are imported while the server is running - since the server will also have the tile database open and access it - and now another process, that the server is not aware of - comes along and changes the file "under the hood" - not good - will lead to desaster, in this case database corruption).

So the question is, do you want to import geo data before you start the server, or do you want to import geo data while the server is running. I assume the latter. It also fits better to what the UI is currently doing and we may use the same API, which means it will be "cleaner" to code.

Therefore, the import has to be a remote operation. locviz is already running as a server and now we run another locviz instance (this time as a client), which will have to connect to the locviz server instance and hand the data to import over to it (via a socket). And then the server instance has to do the actual import. This way, we don't have the database open twice, cause only the server will have it open. And this way, of course, you could also actually do the import "remotely". You could even have locviz on two machines - one where you run the server and another where you have the geo data you want to import. And then you give the "client" locviz instance the IP address or domain name of the locviz server and then it will send the geo data there for the server to import. Of course, you can also do it locally and give "localhost" as the "remote endpoint".

You understand the difference? My assumption is that the locviz server will always keep running in the background - and then you do the import. Then it will have to be a remote operation, i. e. the "import command" must not open the geo database and import the data itself - instead it must hand the data to be imported over to the server instance and tell it to do the import. However, this means you already must have locviz running (in parallel) when importing. Otherwise it cannot do anything.

The difference is important in certain scenarios, for example when scripting things. For example, if the import were a "one-off" operation (server not running) how you would script things would be to import first and then, after the import is done, run the server.

However, I think this is not a good idea. You probably want to keep the server running in the background at all times and then occasionally do imports. Then it has to be a remote operation, because we must not have two processes having the database open at the same time.

This will take some time to implement, but I agree that this would be cool to have. But please confirm again that this is actually the thing you envision. So your assumption when you do the import is that the server always runs, right? It's not an "offline" (no server running) import that you want to do. Cause these would be fundamentally different to implement. But my assumption is that "online" (server running) import is the better way to do it.

[ippocratis]

Personally I don't mind stopping the server and do the one-off import or run a second instance for the purpose .

Most users might use the app on a desktop environment so this solution is for edge cases like remote phone clients as.my self.

If you are interested to add an alternative import to this project I'll be happy to whatever is easier to maintain and makes sense for you the developer .

Thanks for the detailed explanation of how the import mechanism might work .

Thanks for this software .

And Merry Christmas :-)