Ticket #367 (closed: fixed)

Opened 7 years ago

Last modified 10 months ago

Use OpenStreetMap instad of Google Maps

Reported by: stefanb Owned by: kostko
Priority: major Milestone: 3.0b
Component: nodewatcher/frontend Version:
Keywords: Cc:
Related nodes: Realization state:
Blocking: Effort: normal
Blocked by: Security sensitive: no

Description (last modified by mitar) (diff)

In spirit of open source and data openess it would be nice to see this project support map project that uses open source technology and open data - OpenStreetMap. OpenStreetMap if free (as in speech) alternative to Google Maps and maps can be create in combination with z OpenLayers javascript library.

In order to avoid a huge transfer of data Geojson could be used for the transfer of data points in the browser, and status updates without reload the entire page (once we have outgrown http://awmn.net).

Additional layer can also display aerial photos from Google Maps or Bing Maps server (in this case, the visible SSL warnings, so this should not be the default view)

This idea is quite old, he it became relevant with #217 because Google offers HTTPS requests only with payment not for free. But Google Maps v3 (which would be used in a new version can be accessed through HTTPS as well).

Attachments

nodewatcher-openstreetmap.diff (1.2 KB) - added by stefanb 6 years ago.
patch za uporabo OpenStreetMap zemplevidov
nw-3-map.png (601.1 KB) - added by kostko 19 months ago.

Change History

comment:1 Changed 7 years ago by mitar

  • Milestone set to 3.0b

Mogoče bi lahko poskusili dobiti ortofoto podatke. Mogoče bi se lahko povezali z Ljubljanskim geodetskim društvom.

Potem obstaja tudi Geopedia. Ima tudi ortofoto karte in nekako mi izgleda, da poskuša združiti čimveč podatkov. Mogoče bi se lahko z njimi povezali?

comment:2 Changed 7 years ago by stefanb

Če bi Geopedia vključila wlan-lj podatke in jih prikazala na sloju "Brezplačen WLAN dostop" bi to znala biti zanimiva promocija. Dvomim pa, da lahko ponudijo kaj več (niso lastniki podatkov).

comment:3 follow-up: ↓ 4 Changed 7 years ago by stefanb

Aja, pa še disclaimer, da sem povezan z OSM, da ne bo naknadno kake zamere in/ali presenečenj. :)
Z vključitvijo OSM zemljevidov lahko kaj pomagam, vsak pretiran trud okoli zaprtih (pa čeprav so zastonj) podatkov pa se mi zdi odveč (no, razen njihovega osvobajanja).

comment:4 in reply to: ↑ 3 Changed 7 years ago by mitar

Replying to stefanb:

Aja, pa še disclaimer, da sem povezan z OSM, da ne bo naknadno kake zamere in/ali presenečenj. :)

Mislim, da si to že povedal. Ni problema, oziroma še toliko bolje. Je vedno boljše imeti izkušene ljudi.

Koliko pa vas je v Ljubljani?

comment:5 Changed 7 years ago by stefanb

Zemljevid znotraj obvoznice je delo cca 60 urednikov. Cca 10 nas je prispevalo znaten delež tega (v zgodnjih fazah je bilo dokaj enostavno veliko narisati), ostali pa so prispevali kakšne podrobnosti (svoje soseske, popravki napak, dopolnitve...).

comment:6 Changed 7 years ago by mitar

Zanimivo. Ali se je razmišljalo, da bi se preletelo Ljubljano počez in naredilo še zračne posnetke? Z modernim digitalnim fotoaparatom, ki bi ga usmeril navpično navzdol, in dobrim stitching programom bi se lahko naredilo kar nekaj. In se potem umerilo glede na elemente na karti.

comment:7 Changed 7 years ago by stefanb

misliš kot npr http://openaerialmap.org ?
Nekaj sem se igral s tem, a letenje ni ravno najbolj dostopna metoda:
http://labs.metacarta.com/rectifier/map/1343

comment:8 Changed 7 years ago by mitar

Saj zato pa bi recimo zbrali donacije, najeli letalo za panoramski prelet in bi preleteli Ljubljano ter fotografirali ves čas navpično navzdol. Izgleda, da je cena ene ure leta 280 EUR. Verjetno bi se lahko kaj dogovorili. Mogoče se za te stvari lahko najde tudi kakšen razpis. Torej letimo, skozi okno navpično dol obesimo fotoaparat visoke ločljivosti, ki slika ves čas. In to je to. S pilotom pa se dogovorimo, da vozi približno tako, kot da bi sejal ali oral njivo. In potem stitching to zlepi pravilno skupaj (tudi male rotacije in to se verjetno da popraviti). Imamo tudi strokovnjake iz računalniškega zaznavanja med nami, tako da se bi verjetno po potrebi lahko razvilo kaj, kar bi fotografije med seboj porotiralo pravilno in jih zlepilo. Potem pa bi to umerjali.

Torej zanimalo bi me, če bi se dalo komu ukvarjati s tem. Meni se zdi izvedljivo. Najtežji del bo verjetno potem to obdelati.

comment:9 Changed 7 years ago by stefanb

Jp, omenjajo tudi precej nižje cene če letimo sami :)

Če koga zanima lepljenje in kalibracija slik preden to naredimo zares pa lahko prispevam slike in GPS sled iz balonskega preleta :)

comment:10 Changed 7 years ago by lukacu

Mitar je omenjal mene kot t.i. "strokovnjaka" iz racunalniskega vida (ne vem zakaj je govoril o mnozini ... mogoce je se kdo drug) ... samo mene prav nic ne zanima ta stitch-ing ker imam ze dosti drugega za delat. Mogoce kdaj drugic.

comment:11 Changed 7 years ago by mitar

He he, Luka, pa to bi bil super projekt. :-)

Stefanb, kaj te slike so navpično dol narejene? Ker če ne, bo težko.

Sicer pa sem ugotovil, da AutoPano uporablja SIFT ravno tako, kot bi se jaz tega lotil, ko sem sedaj razmišljal, kako bi se. :-)

Sicer pa so tudi navodila za Hugin. Torej jaz bi kar brez GPS sledi vrgel vse slike noter in pogledal, kaj naredi.

comment:12 follow-up: ↓ 13 Changed 7 years ago by stefanb

slike niso bile posnete navpično navzdol, pa tudi če bi hoteli to narediti je definicija navpičnosti iz aviona med letom precej relativna (pač na oko, ker imajo drugi g senzorji preveč motenj). Iz balona je to lažje, ampak z njim ne moreš sistematično prečesati mesta.

Sumim, da nimam dovolj prekrivanj med slikami, da bi Hugin/SIFT znal kaj pametnega narediti. :(

comment:13 in reply to: ↑ 12 Changed 7 years ago by mitar

Replying to stefanb:

slike niso bile posnete navpično navzdol, pa tudi če bi hoteli to narediti je definicija navpičnosti iz aviona med letom precej relativna

Zato sem jaz predlagal, da kamero preprosto obesiš na vrv in skozi okno dol. In bo težnost poskrbela za svoje.

comment:14 Changed 7 years ago by stefanb

Isti pospeški ki begajo tvoje senzorje za ravnotežje bi vplivali tudi na kamero. Pa še veter bi prispeval k temu, da bi se fotoaparat vlekel za letalom na skoraj vodoravni vrvi. :)

comment:15 Changed 7 years ago by mitar

Dobri pomisleki. Ampak saj lahko naredimo fotoaparat aerodinamičen (že to bi bilo dobro zato, da bi bil vedno obrnjen v smeri vožnje), potem pa znotraj aerodinamičnega ohišja, ki bi se vleklo verjetno malo postrani, imaš zaradi svoje lastne teže obrenjen fotoaparat navzdol.

Glede tresljajev in pospeškov pa ... vedno lahko uporabimo jadralno letalo. :-)

Kakorkoli, bomo o tem razmišljali poleti. Samo mene res zanima, da bi kaj takšnega naredili.

comment:16 Changed 7 years ago by stefanb

Zanimivo zna biti še kako bi se izvesek obnašal med vzletom in pristankom (oz. kako ga namestiti in pospraviti med letom) in koliko posluha bi pilot imel za to.

comment:18 Changed 6 years ago by mitar

Verjetno v konfliktu s #689.

comment:19 Changed 6 years ago by stefanb

S samimi zemljevidi ni nujno konflikta, ker je OSM lahko tudi samo kot dodaten (recimo privzet) layer za Googlove skripte. Bi pa v tem primeru vseeno imeli https warninge ("This page contains insecure items"), ker bi skripta prišla po navadnem http protokolu. V primeru OpenLayers skript to ni problem, ker bi jo lahko servirali lokalno, sličice zemljevida pa bi tudi morali servirati z wlan-lj https strežnika (npr caching proxy, da imamo zagotovljene tudi posodobitve)

comment:20 Changed 6 years ago by mitar

Da. Torej dodati layer se mi zdi čisto OK, ker je dejstvo (se motim?), da ima Google super API in da ni primerljive odprte alternative. Torej edina stvar, ki me moti, je ta HTTPS. In ali je to res zadosten razlog, da gremo na OSM? OK, podpirajo odprtosti, to je tudi zelo zelo zelo pomemben razlog meni, ampak na za ceno uporabnosti. Še posebej sedaj, ko bomo podpirali celo Slovenijo?. Kako je OSM drugje po Sloveniji?

comment:21 Changed 6 years ago by mitar

Mogoče lahko zaprosimo njih.

comment:22 Changed 6 years ago by mitar

  • Milestone changed from 3.0b to Next milestone

Changed 6 years ago by stefanb

patch za uporabo OpenStreetMap zemplevidov

comment:23 Changed 6 years ago by mitar

  • Milestone changed from Next milestone to 3.0b

A to je tako enostavno?

Če prav razumem, sedaj pobiramo slike iz njihovega strežnika. Koliko je sploh velikost vseh teh tileov? In predvsem: ali imajo tudi HTTPS strežnik?

Čeprav to že vedno ne reši problema, da se Googlov JavaScript nalaga preko HTTP.

Premikam nazaj v 3.0b, pa bom takrat pogledal ta patch malo bolj podrobno.

comment:24 Changed 6 years ago by stefanb

jp, tako enostavno.

Če se ne omejimo na neko geografsko področje in na neko največjo povečavo je velikost tile-ov precejšnja, njihovo vsakično slepo (dummy) kopiranje na svoj strežnik pa nesmiselno in v nasprotju z dovoljenjem.
To dovoljenje pa vseeno dovoli precej več kot dovoljenja drugih ponudnikov, zato bi nodewatcher lahko bil tudi caching proxy (bodisi na nivoju Apacheja ali Django aplikacije)

Dokler uporabljamo google javascript API (prenešene po navadnem http protokolu) je tovrstni caching proxy smiselen z vidika fair rabe OSM strežnika, ni pa tega nujno potrebno početi.

Če gremo na openlayers.org API, ki bi ga lahko servirali tudi z lokalnega strežnika prek https protokola, bi pa serviranje z lastnega https strežnika odpravilo https opozorila.

Možne so še tehnike pospeševanja nalaganja sličic prek različnih hostname-ov,
a to zahteva wildcard certifikat za *.wlan-lj.net (ali ustrezno več certifikatov).

comment:25 Changed 6 years ago by mitar

In koliko je torej vseh teh? Ker mogoče jih ne bi motilo, da bi imeli še en mirror? Tore ne dovoljujejo le zaradi obremenitve strežnika. Po mojem se itak generirajo na podlagi programa in podatkov, tako da če se dobijo podatki program in se naredi, da se ob vsakem prvem zahtevku stvari zgenerirajo ... Bi to bilo za nas najboljše, ne?

comment:26 Changed 6 years ago by stefanb

Izris lastnih sličic je seveda možno, a po mojem nepotrebna komplikacija (izris, osveževanje podatkov...) in izven obsega tega projekta ... razen če bi potrebovali kakšen projektu specifičen slog izrisa seveda.

Caching proxy bi bil že precejšenj korak (zlasti če bi hoteli vse postreči prek SSL), a to tudi ni nujno.

comment:27 Changed 6 years ago by mitar

Nisi razumel. Ne gre za to, da bi mi potrebovali lasten izris, ampak da bi lahko ponujali slovenski mirror za OpenStreetMap. Recimo da tudi o tem razmišljam. Če praviš, da to ni potrebno, super.

comment:28 Changed 6 years ago by stefanb

Aaa, to je pa druga zgodba (oz. projekt :).

Mirror bi seveda bil dobrodošel, a v tej fazi to ni kritična komponenta nobenega od teh projektov.

  • Za wlan-lj bi bilo dobro, ker ta vir omogoča večjo prilagodljivost, vključno s ponujanjem ploščic prek SSL, a to se da doseči tudi s caching proxy-jem.
  • Za OpenStreetMap pa, ker bi se morda še kdo odločil za njegovo uporabo in bi s tem dobil še kakšnega urednika, ki bi vrisal svojo ulico. A slednje se (zaenkrat) da doseči tudi s strežnikom v tujini.

Dokler nimamo bolj tehtnega razloga za komplikacije bi se držal načela KISS in izkoristil simbiozo, ki jo projekta lahko živita.

comment:29 Changed 6 years ago by mitar

Glej, mene zanimajo facti. Koliko to zaseda, koliko CPUja kuri, kako težko je to vzpostaviti.

Razumel sem tvoje stališče, strinjam se z argumenti, ki jih navajaš, le manjkajo mi še nekateri argumenti, ki bi jih prosil, če jih lahko pripraviš. Potem lahko komaj potegnemo sklep. Ker mogoče so zahteve preprosto takšne, da tega ne moremo izpeljati, tudi če bi si želeli. Ali pa so tako smešno majhne (namestiti Debian paket), da je to res enostavno izpeljati.

Zato jaz ne bi rad vlekel prehitrih zaključkov, če se nismo zbrali vseh podatkov. Se pa torej seveda strinjam: ni kritično, gre doseči drugače, v Sloveniji verjetno ni toliko povpraševanja za tem (ampak saj naš mirror bi lahko uporabljali tudi iz tujine), KISS je pomemben. Ampak ključnih podatkov, takšnih numeričnih, ki jih je zelo enostavno potem ovrednotiti, pa še nisi podal. In jaz sem te vprašal le za to, mogoče celo malo iz radovednosti, da malo več izvem o sistemu, da si lažje predstavljam o kakšni pošasti sploh govorimo.

comment:30 Changed 6 years ago by stefanb

Splošen postopek je opisan na
http://wiki.openstreetmap.org/wiki/Creating_your_own_tiles

Performančno najbolj ugoden je Mapnik, ki uspe izrisovati ploščice po potrebi.
http://wiki.openstreetmap.org/wiki/Mapnik
Tu so naštete tudi nekatere strojne zahteve.
Za občutek kakšen strežnik ima samo za ta namen osm.org: http://wiki.openstreetmap.org/wiki/Servers/yevaud :)

Za konkretno ilustracijo prostorske zahteva za samo zoom 18 (na katerem je že vidno na kateri fasadi zgradbe je antena), kjer je svet razrezan na mrežo 218 * 218 ploščic povprečno velikih 200 bajtov:

((218)2) * 200 bytes = 12.5 terabytes

  • -mnogo ploščic se podvaja (70% je morja): *.3
  • +vsi zoom-i od 1-17 vzamejo še enkrat toliko prostora: * 2

(((2^18)^2) * 200 bytes) * 0,3 * 2 = 7,5 terabytes (ob optimalni velikosti clustra)

comment:31 Changed 6 years ago by mitar

Super. No, potem je stvar itak jasna. Trenutno to sploh ni izvedljivo, tudi če bi si želeli.

comment:32 Changed 6 years ago by mitar

  • Effort set to normal

To je zanimiva rešitev: Mapstraction.

comment:33 follow-up: ↓ 34 Changed 6 years ago by mitar

Dajte vsi starajte ta issue.

comment:34 in reply to: ↑ 33 Changed 5 years ago by stefanb

Replying to mitar:

Dajte vsi starajte ta issue.

It helped, their issue is closed as of today.

Google now allows access to maps API (and tiles) over SSL to anyone:
http://googlegeodevelopers.blogspot.com/2011/03/maps-apis-over-ssl-now-available-to-all.html

comment:35 Changed 5 years ago by stefanb

FYI: Google is charging for excessive maps API usage:
http://code.google.com/intl/uk-UK/apis/maps/faq.html#tos_pricing

Up to 25,000 map loads per day is still free though:
http://code.google.com/intl/uk-UK/apis/maps/faq.html#usagelimits

Do we have any info where are we at with the number of loads currently?
(my guess: < 1000 / day)

comment:36 Changed 5 years ago by mitar

Non-profits and applications deemed in the public interest (as determined by Google at its discretion) are not subject to these usage limits.

In addition we recommend that eligible Non-profits apply for a Maps API Premier license through the Google Earth Outreach program. This provides a number of benefits, including the right to opt-out of advertising, higher quotas for Maps API web services, and technical support.

But probably this is only for USA non-profits.

And I would just wait until they contact us: ... a warning may be shown on your map and a Maps API Premier sales manager may contact you to discuss your licensing options.

By quick check we had around 500 loads yesterday.

Thanks for pointing this out.

comment:37 Changed 4 years ago by mitar

Last edited 4 years ago by mitar (previous) (diff)

comment:38 Changed 3 years ago by valentt

I'm talking with guys from OpenStreetMap-hr to reboot this idea and get Google map replaced with OpenStreetMap if possible...

comment:39 Changed 3 years ago by mitar

What do you mean? It is easy to integrate OpenStreetMap into Google API as a layer. Just we haven't yet done it.

comment:40 Changed 3 years ago by valentt

OK, I read this ticket but misunderstood it as stuck because it passed over three years and nothing happened so I though it was not easy thing to do and that you need help.

If it is easy how come after three years it is still not implemented?

comment:41 Changed 3 years ago by mitar

Because there are other more important things. Map works and we are concentrating more on things which do not even work. We will get back to things which work when we finish 3.0.

PiplMesh has already an implementation with OpenStreetMap.

comment:42 Changed 3 years ago by stefanb

Can we be sure that referenced Google's javascript and images aren't and won't be setting any cookies?

comment:43 Changed 3 years ago by mitar

We probably cannot. And we cannot even for Disqus. Or YouTube embed (for this one there is a special domain which you can use so that cookies are not set).

comment:44 Changed 3 years ago by kostko

  • Component changed from nodewatcher/core to nodewatcher/modules
  • Milestone changed from 3.0b to Next milestone

comment:45 Changed 3 years ago by valentt

Is current map part of Nodewatcher code or separate code?

Would it make sense to have separate javascript mar that uses OpenLayers so it can be embedded on any page.

How do you get list of nodes for specific project? Is there an API for getting list of nodes and their geo location?

comment:46 Changed 3 years ago by valentt

  • Description modified (diff)
  • Summary changed from Uporabiti OpenStreetMap zemljevide to Use OpenStreetMap instad of Google Maps

comment:47 Changed 3 years ago by valentt

  • Description modified (diff)

comment:48 Changed 2 years ago by mitar

  • Description modified (diff)

comment:49 Changed 2 years ago by mitar

Is current map part of Nodewatcher code or separate code?

Part of nodewatcher code.

Would it make sense to have separate javascript mar that uses OpenLayers so it can be embedded on any page.

What do you mean? To allow embedding maps? This can be done with Google Maps as well. But yes, allowing embedding map would be a great feature.

How do you get list of nodes for specific project? Is there an API for getting list of nodes and their geo location?

For a project you cannot, but all active nodes can be accesses as GeoRSS. See at the end of main nodewatcher page.

comment:50 Changed 2 years ago by stefanb

Leaflet JavaScript library has become popular in the mean time, replacing OpenLayers. See http://switch2osm.org/using-tiles/getting-started-with-leaflet/
Also GeoRSS is slowly being replaced by GeoJSON, which could be offered to other sites and applications, either to include in their map od in their database for futrher processing.
http://leafletjs.com/examples/geojson.html
http://geojson.org

comment:51 Changed 2 years ago by mitar

There is also Leatlet-hash to store position of the map in the URL.

comment:52 Changed 2 years ago by valentt

many communities are starting to use libremap (Freifunk, RéseauLibre, and so on).

comment:53 Changed 2 years ago by mitar

We could definitely export data in a format compatible with it so that nodes can be seen there as well. Or mirror node data in CouchDB as a module in nodewatcher. But this they have a bit different goals than nodewatcher. Anyway, that would be a different ticket. Or we could have a module for nodewatcher using it to display a local map.

comment:54 Changed 2 years ago by valentt

I don't know enough about Nodewatcher to understand what you mean by "they have a bit different goals than nodewatcher", but if you could work with libremap team and make it usable with nodewatcher than this seams to me like a nice fit, especially if everybody else will also be using it.

comment:55 Changed 2 years ago by mitar

I don't know enough about Nodewatcher to understand what you mean by "they have a bit different goals than nodewatcher",

nodewatcher tries also to generate firmware images and monitor nodes. libremap seems just a map of nodes with static information for each node.

comment:56 Changed 2 years ago by kostko

Libremap is something completely different. It is not a map widget, but it is a system based on CouchDB where nodes (or a system like nodewatcher) push information to a server (or a set of servers) that can then synchronize this database among themselves. Then a GUI can display all the collected information on a map. It is a very simple thing.

So yes, nodewatcher 3 can easily support a module that pushes this information to libremap. Implementing such a module should be fairly easy. But this has nothing to do with using OSM instead of Google Maps in nodewatcher itself.

comment:57 Changed 2 years ago by mitar

And module could also embed a map so that you could have on nodewatcher a "global map" and "local map". But, on the other side, it is true that having too maps is a bit confusing. It would be useful that when you zoom in you see all links and so on, but if you zoom out and see the whole world, you could see all other nodes from all other networks as well. There are many ways how to do that. For example, having a layer being displayed from Libremap. Or we could even read ourselves data from there. On the other side, it would be interesting if we could reuse their codebase. But I do not think they support drawing links between nodes?

comment:58 Changed 2 years ago by kostko

On the other side, it would be interesting if we could reuse their codebase.

Well, their codebase consists of a CouchDB server and everything is then handled client-side if I read the source code correctly (I did it really quick so I might be wrong). So, to include other stuff in our "global map" we just need to design this global map with hooks/API in mind, so another module can provide additional points on it.

And this additional module would then do a really simple CouchDB query to get all the nodes from LibreMap. Libremap is really simple.

comment:59 Changed 2 years ago by mitar

I was thinking about client side code reuse. So how they draw nodes/do map. Maybe we could use their code and push our nodes into their map, rather than reimplementing drawing and then be pulling nodes into our map.

comment:60 Changed 2 years ago by stefanb

OSM is moving towards offering its tiles over https:
https://lists.openstreetmap.org/pipermail/talk/2014-February/068988.html

Hi OSM Talk,


Approximately 20% of visitors to OpenStreetMap.org today will receive
the default tiles via HTTPS (ssl/tls).


This is to test HTTPS on our CDN infrastructure.


If anyone experiences any problems please can you report them in
channel #osm on http://irc.openstreetmap.org/


Technical:
We use a dedicated wildcard *.tile.openstreetmap.org certificate
signed by RapidSSL/GeoTrust on the 12 geo distributed cache servers.
The SSL termination is handled by nginx. SSL Report:
https://www.ssllabs.com/ssltest/analyze.html?d=a.tile.openstreetmap.org
Enabling SSL on tile is a step towards being able to offer HTTPS as an
option on openstreetmap.org for more than just the login page.


Kind regards,

Grant
Part of OpenStreetMap sysadmin team.

comment:61 Changed 20 months ago by kostko

  • Component changed from nodewatcher/modules to nodewatcher/frontend
  • Milestone changed from Next milestone to 3.0b

I agree that all map-related code in nodewatcher v3.0b should use Leaflet instead of any vendor-specific code.

comment:62 Changed 19 months ago by kostko

In 2c9b6331be0f5a2de0372c2dcac76d2b224f7989/nodewatcher:

Use Leaflet with OSM instead of Google Maps. See #367.

comment:63 Changed 19 months ago by kostko

Nodewatcher v3 now uses Leaflet with OpenStreetMap in the node configuration editor. The next step is to create the global map module that displays all the nodes together with the topology on a single map.

Topology information will come from datastream (same as in #736) and there are two ways to obtain location information:

  • The location module may store the location into datastream (by registering a topology storage extension), which would also (later) enable view of node location changes through time (although this is rare).
  • Location information may be fetched from the database. In this case only the current information will be available.

Opinions?

comment:64 follow-up: ↓ 65 Changed 19 months ago by mitar

The location module may store the location into datastream (by registering a topology storage extension), which would also (later) enable view of node location changes through time (although this is rare).

Would that be a separate datastream then? So a datastream for a location of a node? Wouldn't then we make a datastream for every configuration option as well and have it stored through time?

On the other hand, having location in would finally make mobile nodes really mobile and you could trace how they move. Or maybe if we don't want datastream for each configuration option, we could have location datastream only for mobile nodes?

BTW, maybe we should think about datastream being versioning for registry. So we store monitoring into registry, which we then copy into datastream. And why not do the same with configuration? Only that configuration we would make an entry only when configuration changes, not every time.

The question is: do we have a datastream for each value in configuration, or do we have one big datastream for all configuration.

comment:65 in reply to: ↑ 64 Changed 19 months ago by kostko

Replying to mitar:

Would that be a separate datastream then? So a datastream for a location of a node? Wouldn't then we make a datastream for every configuration option as well and have it stored through time?

No, this would be stored into the topology graph as a node attribute, so it is synced with the topology.

comment:66 Changed 19 months ago by mitar

No, this would be stored into the topology graph as a node attribute, so it is synced with the topology.

So you have an extension point where various modules can add node attributes into topology graph? So modules add ETX onto links, and location onto nodes?

But we should think about other ideas I made as well.

comment:67 Changed 19 months ago by mitar

(Maybe in some other ticket.)

comment:68 Changed 19 months ago by kostko

Yes, modules can add node and/or link attributes into the topology graph.

I agree that other ideas are also possibilities, but in this case fetching data would require several API calls, querying for the values that were in effect at specific points in time. This would also enable inconsistent views on the data – also what happens when a node's streams are removed?

comment:69 Changed 19 months ago by mitar

I agree that other ideas are also possibilities, but in this case fetching data would require several API calls, querying for the values that were in effect at specific points in time. This would also enable inconsistent views on the data – also what happens when a node's streams are removed?

I don't think there is anything much wrong with denormalizing data for common use cases. So having location and ETX in topology datastream together with having a separate datastream for ETX through time and location through time.

What I am saying is that we should have two ways to fetch data:

  • through a relation database you can make complicated queries and fetch latest data (monitoring and configuration and anything else)
  • through a NOSQL database you can make simple request for data and fetch any data in the past, and as it is common in NOSQL world, we denormalize sometimes for common cases, but in the worst case you have to make multiple queries and you might have to reconstruct a consistent view yourself (because in most cases slight inconsistencies will not matter)

Most modules will use only the former, but some modules currently will use the latter. And then we can have more specific or generic modules later on for the latter.

comment:70 Changed 19 months ago by kostko

Ok, I believe that this second part should be moved into its own ticket and leave here only things related to the API that we will currently use for the map. Otherwise, I agree that there should be some kind of support for having historic view of configuration values.

comment:71 Changed 19 months ago by mitar

Ok, I believe that this second part should be moved into its own ticket and leave here only things related to the API that we will currently use for the map.

Yes. So as I said, I think it is OK to have denormalized data so store location data into topology.

comment:72 Changed 19 months ago by mitar

I made #1227.

comment:73 Changed 19 months ago by kostko

In de32dd553c1c2d74414a9dd934316f13dd5e817d/nodewatcher:

Implemented simple but extensible global map. See #367.

Changed 19 months ago by kostko

comment:74 Changed 19 months ago by kostko

  • Status changed from new to closed
  • Resolution set to fixed

We now have a global map interface.


It is a simple Leaflet-based widget, currently with extension points similar to the topology rendering extension API (#736):

  • Datastream topology storage extension mechanism is simply reused so all stored attributes are available. The location module adds storage of a geographical location attribute.
  • Modules can extend map rendering by registering specific partials which contain Javascript-based extensions. For example, the above image uses two extensions:
    • The administration.location module registers an extension that creates markers for all nodes which have a location set. It also creates polylines for all topology links where their source and target have a location set.
    • The routing.olsr module registers an extension that changes the color of the polylines based on the stored ETX metrics of the links.

comment:75 Changed 19 months ago by kostko

Note that the markers are currently ugly defaults, but using the extension mechanism described above, a module (for example, the wlan slovenija skin) can also style the markers however it wants.

comment:76 Changed 19 months ago by stefanb

nice progress!

btw, some remaining massive white areas in the map will be improved in the near future as we got permission for import data from ministry of agriculture:
http://forum.openstreetmap.org/viewforum.php?id=58
some more details:
https://wiki.openstreetmap.org/wiki/Slovenia_Landcover_Import_-_RABA-KGZ

comment:77 Changed 19 months ago by kostko

Great :-) I have also contributed some updates around Kranj and will be fixing some more.

comment:78 Changed 19 months ago by valentt

Awesome! Will OpenStreetMap be applied also to current node map or will it be only in future when you upgrade to NW 3.0?

comment:79 Changed 19 months ago by kostko

Sorry, there will be no more changes in version 2.

comment:81 Changed 10 months ago by stefanb

FYI, some areas in Slovenia already got the import of farmland into OpenStreetMap, which improves the map in rural areas by orders of magnitude.

Example: http://www.openstreetmap.org/#map=14/45.5404/13.8390

Note: See TracTickets for help on using tickets.