Chart caching for Freeboard SK
I am wondering how freeboard sk saves the charts from online sources, like openseamap and openstreetmap. Looking into this since I will have internet for only a couple hours a day for a trip and want to know if suddenly these charts will disappear.
126 Replies
Also wondering about the active captain plugin, and if those will disappear if I have no internet after a while. I see the cache time config, but I am unsure if that has to do with the fetching or display of data
Freeboard-SK does not cache charts, instead it uses the chart metadata provided by the chart / pmtiles plugins to fetch chart tiles.
Both the charts and PMTiles plugin will serve chart tiles from local storage, for other chart types you will need a chart cache e.g. GadrielCache or equivalent.
FSK supports WMS and WMTS sources so any cache solution or map server should suffice.
What about resources provider? Does that store the charts from signalk chart provider locally? Currently I just have openseamap and openstreetmap, which are both from the web. When I shut down my internet it seems like the tiles I have looked at are still saved, will they ever become unavailable in that scenario. And in addition to this, I believe my active captain resources (which are just notes) go through resource provider as well. Will these follow the same process?
I have NOAA as a WMTS server source in freeboard sk, which does not cache at all. But it seems charts plugin -> resource manager -> freeboard sk does cache charts somewhere. I would like to use the WMTS source within charts plugin but how do I use the server within charts plugin?
The chart metadata for WMS, WMTS and tilejson added via the FSK chart list is persisted by the resource provider. Not the tiles though
the chart tiles tah you have loaded (looked at) may get cached by your browser
but afaik nothing in the SK server provides tile data caching (storing for later offline use after being fetched) nor seeding (pre-fetching data to be cached, even if you have never opened the particular tiles on your browser)
you need to run something like MapProxy or GaladrielCache to do that
I run a mapproxy instance in docker, works like a charm
I run a small python server which acts as a proxy to the map services I want to reach. Works nicely
a chart cache e.g. GadrielCacheGaladrielCache
The current problem with those is I have signalk server running in venus os on a raspberry pi,so I have no regular Linux OS stuff like docker or otherwise. It should also stay that way which is a little annoying but for power and cost its important.
Thanks for the answers though, maybe theres a proxy npm package for signalk
VenusOS includes PHP. Specially for GaladrielCache.
Ah ok thanks, I’ll look into it
Just asking out of curiosity, tile map caching seems to be a popular request, Is there a particular reason to why we dont have addin for this in signalk already?
I can think of legal reasons, but nothing else comes up to my mind
Not by policy, just that nobody has contributed a solution yet
There are also tile sets available in mbtiles format, notably NOAA, so just a single file to download instead of seeding them one by one
Maybe most people use opencpn, lots of mbtiles on the web & you can make your own. https://hackingfamily.com/Cruise_Info/Equipment/Chart_Downloads.htm
Here is a link with "how-to" for off-line charts.
https://pysselilivet.blogspot.com/2021/09/signal-k-charts-offline-online.html#more
OpenCPN is great software, but i hate that i have to use it through VNC or RDP from my tablet or phone which gives a very clunky feeling for me. And that is exactly what i love with Signalk and FreeboardSK, i can see my maps from any device with a browser. And to combat this i made myself a python server,( I never work with JS/TS thats why i choose python), which gives me access to maps from google maps, cmps, nav and skippo. Looking through the code of SAS Planet helped to understand how things work with the map providers a little bit
I can download entire region maps based on regions defined by geojson, but in the end i need to spawn a different service, and the whole setup is clunky and requires some manual fiddling etc.
Having seen this issue, people asking for cached map tiles come up over and over again, i was just wondering why no solution already exists for it as an addin, and also legality of downloading tiles from navi, cmps and skippo is a grey area for me, hence my hesitation in making the code publicly available
Yesterday, I asked the AI overlords how this can be achieved in signalk as an addin and the offered answer was short and simple, perhaps i can get my hands dirty with some JS code and create my first addin 😄
👍if i were you i’d pick ts over js, that’s what i do
And always happy to give my feedback on solution arvhitecture - sooner rather than later
phone is horriblle but the android opencpn app works well. VNC bit clunky but still useful as that's where the radar display is. Enjoying vscode with Ai writing the code these days, often works first time or just keep banging the error messages in til it does. python so far but must try an js sigk app soon! https://discord.com/channels/1170433917761892493/1402677564593537095
It would be great if we can find someone whom wants to build a SK/Freeboard integrated chart provider that offers a simple UX for the commoners and regular mortals. I think it’s one big barrier to more adoption. Charts are a headache.
I can’t answer because I don’t know what it is or does. Thanks for sharing. I’ll look into it.
But that statement I’m just making support my claim - if folks like us whom develop the platform need to ask about if and how. Imagine for common folks. It’s not obvious enough and too complex so people go to OpenCPN instead because in the end, what’s a chartplotter without charts.
I mean, even if we had a commercial chart vendor write a chart download plugin it would be better than nothing.
I honestly think that with simple out of the box chart integration, we have a much more accessible and flexible solution then most commercial vendors on the market.
I wonder how AVNAV does charts!?
what’s a chartplotter without chartsGaladrielCache is a part of GaladrielMap suite, not of SignalK (however the GaladrielMap chartplotter is available in the SignalK). And the GaladrielMap has no problems with the maps (mostly, yeah).
I wonder how AVNAV does charts!?https://o-charts.org/ But the o-charts rights to the charts are the same as mine, but I don't sell charts.
You can use e.g. QGIS to create off-line version of on-line charts and copy them to media available on your boat.
I have done so with success, but it is a painfully slow process...
Screenshot of one of the parts I have created, for the higher zoom levels I have created smaller areas due to the slow processing up to 15 works reasonable....
You can use e.g. QGIS to create off-line versionWhy?
I used s57 conversion tool. I mean it's a great tool, no doubt, but it's not something most people can do. Maybe we should see if we can integrate ochart!
There was some work going on in that space https://discord.com/channels/1170433917761892493/1354213962430025790/1360180328571998422
I have a proof of concept implementation, lets discuss 🙂
I created a basic SK addin which adds an REST endpoint like /plugins/signalk-mapproxy/eniro/9/285/153.png, if the file already exists in the cache it returns the image otherwise it fetches it from the remote and stores in cache
on the freeboardsk side, i add a new provider with the url http:// myipaddress:3000/plugins/signalk-mapproxy/eniro/{z}/{x}/{y}.png
Seems to work nicely and i can switch easily between googlemaps, cmap and eniro.
it would be supernice if the chartsprovider would pick these maps automatically,
also be nice if i can select a region/route in freeboardsk and be able to download the tiles with desired zoom levels to the cache as a long job that works in the background
Happy to help with the chart provider work… some kind of dynamic signalk chart metadata publishing would be nice
🙂 just saying there are more ways to accomplish an off-line versions of on-line maps.
one way to do this would be to add the proxy functionality to the charts plugin
but a caching proxy is kinda easy, seeding/prefetching so that end users can use it is more complicated
You found the most tricky one. There is also a MOBAC and SAS.Planet, if you don't like the GaladrielCache so much.
Have not tried Galadriel, but a big LoT fan so really like the name, also a beutiful ship/boat name 🙂
BTW found QGIS quite straight forward 🙂
I looked through the chart provider source code, I guess what can be done is to add a cache checkbox to the configuration and handle those charts similar to the "Chart paths" charts; create an endpoint and serve the tiles through that endpoint, while sending the client the signalk server ip address and the endpoint as tile source
I wrote myself a server for ochart tiles, but that's definetly against ToS, so I'm afraid it cannot be shared.
Anyway, as long as OpenCPN is installed, with o-charts plugin, and a valid license, it's realtively "easy" to fetch the tiles from OpenCpn. That's what AvNav does, and I think that's legal
Oh its doable for sure but the discussion was on how to increase adoption by non-dev/by regular sailors. Having to install other non-sk related apps, run a proxy and configure custom servers is not something most can do, or are willing to do. I think it need to work out-of-the-box, with simple buttons to add/buy charts from within Freeboard. It's why tablet navigation apps are so popular. Also a main reason why OpenCPN is a go to solution. I heard OpenCPN plugins are the second factor for it's popularity but, without charts to start with and get a crowd, who would build plugins...
Having to install other non-sk related apps, run a proxy and configure custom servers is not something most can do, or are willing to do.If you don't want to use what you have, pay for the development of what you want. Personally, I don't care if there are maps in the SignalK, but I can write an analog the GaladrielCache for the SignalK. With GIU and working out-of-the-box.
Hi @Teppo Kurki
I modified the code for the charts-plugin adding support for caching online maps, if you like to give some comments before i create PR, here is the link
https://github.com/batuakan/charts-plugin/tree/caching_online_map_sources
Github pull requests for commenting them, so please go ahead and open one
I recently built a galadriel-cache/galadrielmap docker container from the ground up, because i like to take the hard road. 🙂 very happy with the result and having navionics+cmap+google maps+offline caching is great.
but i guess having just done this is most people don't want to run yet another "thing" besides signalk and add to their administrative workload. i have to now run galadrielmap (php), galadrielcache (php), signalk (node), plus whatever plugins i'm running. i think reducing the administrative overhead would be nice and just being able to define the charts in signalk and having them cached would reduce administrative overhead a lot.
happy to share my dockerfile for galadrielmap/galadrielcache if it's useful to folks.
i think it would be awesome if there was a galadrielmap/galadrielcache image on docker.io that could be a one click install. I know you have the binary docker image release on github, but that's a less common path. 🤷♂️ docker pull xxx is the common route, i guess.
my update to the chart provider plugin will bring eniro+cmap+googlemaps with caching to signalk. navionics is bit more tricky as it runs on a time based access token which must be acquired from the navionics server and i don't want to put in navionics specific code in the code base.
I wish there was a way to have custom scripts uploadable to the charts-plugin so for every tile provider we could create custom fields such as navionics access token beyond the x,y and z values.
Beyond the changes in charts-plugin, we also need a way to pre-fetch tiles from the remote server, e.g download tiles for this region. and a way to remove the tiles from the cache e.g. remove the files for this region. If freeboark-sk takes on this load things would be more integrated, otherwise it would be another webapp for signalk
@night199uk btw, I would love to see your dockerfile for GaladrielMap/GaladrielCache
prefetching should not happen by a client, the plugin should do it
there is already code that does search and replace for ~tilePath~, something similar would work for dynamic url params. there aren't that many use cases for that, so some hard coded patterns would be ok in my book
the charts-plugin creates a proxy and an endpoint to fetch the tile, there should be another mechanism that says given a region and list of zoom levels, here is list of million tiles i want to fetch, go fetch them. I think this is above the charts-plugins functionality. Yes a plugin should do it, but it should not be the charts-plugin in my opinion.
For navionics, you create a timestamp from unix time and query a different navionics endpoint to get a token which expires in 6 hours, and you append that token to your navionics tile url. We include that token endpoint in the charts-plugin and provide another variable like {navionics_token} so a user can include it in their tile url, but that would be a maintenance nightmare every time navionics changes how that token is calculated, but also a legal grey area.
@Teppo Kurki what does ~tilePath~ do? I saw it in the code but didn't understand what it does.
Imho seeding is exactly in chart plugin’s domain. Think about it from the user’s perspective: why would they need to install and configure multiple plugins, when all they want is ”this map for this area available offline”?
As navionics i was naive. That would fit the chart resource provider structure, as a separate plugin, and charts plugin could just provide caching
but shouldnt there still be a UI somewhere, where the user draws a region, and tells the sk server, i want to start a concurrent multithread job to fetch the tiles for that region, something still needs to monitor that job, cancel it if needed, there could be multiple jobs running.
How do you propose to do it in the charts-plugin?
a plugin can be also a webapp and its webapp can modify the plugin's configuration. if you could do it in another plugin why would it not be possible in charts?
a plugin can also provide a custom configuration screen instead of the default one that is generated from the schema, making it easier to find in the plugin configuration context
as for the seeding background jobs that complexity is the same, no matter where that logic resides
Actually not - if the seeding is someplace it needs to use the caching configuration, so having it in the same place makes things easier
Just want to say, if this is succesful then SignalK is taking a giant leap forward, so grateful for your efforts!!
IF needed I can assist with testing.
The GaladrielMap docker is originally intended for a running demo. In general, the GaladrielMap suite is designed to work on weak computers where docker is inappropriate. I'm not sure if it's necessary to distribute the GaladrielMap suite as a container as well. It's quite troublesome.
in my wishlist 😄 it would be awesome if this downloader UI was just in freeboard SK. i can envisage a button on the left for "download tiles" for the current view. already i have a few different mapping webapps based on signalk (signalk-trackviewer, freeboard-sk, anchor alarm, galadrielmap) all of which have different map/UI implementations and end up looking at different sets of charts.
It sems like it would be not hugely difficult to add the button to the freeboard-sk UI and do something like galadrielmap does to display that tile boundaries and allow selection?
e.g. my current wish / thing i would like to work on when my boat stops moving is to see if i could get a PoC of the trackviewer in the freeboard-sk so I can see my tracks on all the same charts providers that freeboard already supports. the trackviewer is amazing but it has a limited set of charts compared to freeboard.
I agree with you and i agree with Teppo as well. At first i thought the charts-plugin should only be responsible for fetch and delivering tiles, but we can put additional logic in there to pre-fetch regions drawn in freeboard sk, manage the jobs and delete cached tiles based on selected region again. and once we have all that logic, i dont thing it should be difficult to add a button in freeboard sk to the popop that shows when you select a region to show "Download tiles" and then charts-plugin should go and fetch the tiles from remote server, you can go to web view of the charts-addin to see the progress and pause or stop the download, as it usually requires a lot of time
I agree with you on your second message as well, different functionality uses their own webui with their own maps, i love the trackviewer plugin, but i would like to be use it with my online maps rather than the built in map. In my opinion it would be nicers if all chart plotter related tasks are handled from with in freeboard sk. Perhaps there could be a contribution point for addins; giving us 3 sorts of addins bakend, webui and freeboardsk addins
@Ba2 stole my thunder - we can start with for example simple, manually entered seeding bounds in the plugin config, then add a map based selection mechanism and move on to freeboard-embedded seeding configuration
it would probably be also easy to add track layers off the charts api to trackviewer, ping @Copprhead
Yep, using the charts provider charts in TrackViewer has been suggested in the past and I would like to have that, too. 🙂 Let's see how cold the winter gets, maybe I find the time.
Being able to use cached online maps from all the interesting places would really motivate me to build this feature 😉
Can you write the full URL? I'd like to try it.
It won't work, it points to a server i wrote myself which, as I said cannot be shared for legal reasons
Ok, got it.
If you feel like spending some time, install avnav and it's ochart plug-in, and check the url it uses to query the charts
Just realised that i indirectly implicated that the author of a piece of opensource sw is the only person to add features - anybody and everybody can contribute!
I have added a new endpoint to seed maps based on region from freeboardsk, however i am completely stuck with adding embedded webapp to charts-plugin. I couldn't even get a hello world page to show up. Anyone dare to help?
I am happy to help...I will have a look at the PR.
I am currently adding support to Freeboard to select an area for use as an entry / exit alarmed region, so I can extend the "selection" mechanism for use to request map tiles.
Start with a regular webapp, embedded is just polish. And i can help of course
Getting closer 😄
@Teppo Kurki I think i am done, I updated the PR.
I am sure there are more things to fix, but lets wait for the review for that 🙂
For anyone else who want to test and give feedback here is the repo: https://github.com/batuakan/charts-plugin/tree/caching_online_map_sources
so: is the idea that the user first creates the Region resources and uses them? or is this just the version? I would think just entering two points with lat lon coordinates would be easiest to start with?
user creates a region in freeboardsk, and then moves to the charts-addin webapp, selects the region, selects which map to seed and max zoom, I thought this was a good way to keep an eye on active downloads, however the actual download can be started from freeboardsk in the future in this screen
Also fetching a million tiles from a remote server is somewhat slow, excluding any areas not of interest is a big time save.
I was thinking I could add a “drag box” to select an area and submit it to the charts plugin to define the area required.
for a user it would be useful to be able to follow the trail from plugin install to plugin config to webapp to Freeboard => add documentation and links to relevant places
Just for my clarity...assuming the plugin exposes endpoints to facilitate triggering a download from FSK (or other app)... the process for the app to follow would be:
- Retrieve a list of the chart types (i.e. Eniro, OpenStreeMap, etc)
- Collect box coords from the user e.g. using bottom left, top right format
[[-7.123, 54.456],[-7.123, 54.456]]
- Allow the user to make a selection from the chart types
- Submit the selected chart type & box coords
Is this the proposed sequence?The way it now works:
- configure maps in chart plugin with proxy enabled
- create regions in Freeboard
- create seeding job in charts plugin’s webapp by entering (chart provider, region uuid, layer depth)
I was querying whether we could replace using regions with a bounds submission.
Kind of what i wrote above: entering two coordinates. We could have both, region and bounds
Simple enough: PR the PR 😆
Combining both of your answers,
Pre configuration:
- Create proxied online chart provider
- Create regions in freeboardsk
Seeding
- create seeding job in charts plugin’s webapp by entering (chart provider, region uuid, layer depth)
The endpoints i added are
POST: /cache/:identifier/:regionGUID/:maxZoom
DELETE: /cache/:identifier/:regionGUID/:maxZoom
GET: /cache/progress
PUT: /cache/progress/:id/cancel
for downloading a region is fetching matching its uuid and a chart is selected based on its provider id. Charts-addin webapp fetches those automatically.
I like sending in the region because they are well defined structures, and in the future we can have the option to create mbtiles from the cached tiles, using the regions name, bounds etc.
Why dont you like regions?
@Teppo Kurki In the meanwhile regarding our earlier conversation with custom access tokens that expire frequently and tokens that must be appended to the remote url with some certain map providers: What do you think of having js addins to the charts-addin, you leave your custom ChartProvider js code to the ~/.signalk/charts/ directory and they will be picked up automatically when the server starts and be added to the list of charts providers. some properties such as headers or remoteUrl can be getter functions so they can be calculated on demand.
I made a PoC implementation and managed to get the difficult chart provider to work
that would work, but is....not very user friendly. i think one major goal here would be making chart downloads for offline usage easier. easy = install & configure a plugin
one way to do this would be to package the chart addin as a plugin and on start emit a PropertyValue with for example name 'ChartProvider' and the value would be the provider
all the user would have to do is to install the other plugin, in addition to the charts plugin
see https://github.com/SignalK/nmea0183-signalk?tab=readme-ov-file#custom-sentences for similar feature where a plugin can contribute a custom NMEA0183 sentence parser
GitHub
GitHub - SignalK/nmea0183-signalk: NMEA0183 to Signal K parser
NMEA0183 to Signal K parser. Contribute to SignalK/nmea0183-signalk development by creating an account on GitHub.
That is, if you do not expect users to create maps themselves.
is the idea that the user first creates the Region resources and uses them?😁
Does the cache work with wmts servers?
I see the mbtiles case, does that allow for wmts providerswhen they are input into freeboardsk?
If the question is addressed for my ongoing cache implementation, i dont know about wmts providers, I googled a bit but couldnt find one to setup to test, if you send me your charts config I can try
I go into fsk, go to charts, add a chart, png, WMTS, this is the URL: https://gis.charttools.noaa.gov/arcgis/rest/services/MarineChart_Services/NOAACharts/MapServer/WMTS
I couldnt get it to work with SignalK, but i was able to add it to the map list in QGIS. WMTS maps use a different way for fetching the tiles and they don't go through the same tiling process as the tilemaps, so i guess the cache wont work for WMTS maps
No. WMTS is a Web Map Tile Service. At you url is a ResourceURL parm - use it.
so it goes something like this
..../MarineChart_Services_NOAACharts/{Style}/{TileMatrixSet}/{TileMatrix}/{TileRow}/{TileCol}.png
what do you fill in for the Style TileMAtrixSet TileMAtrix TileRow and TileCol?
tileRow might be {y} and tileCol might be {x}, or vice versa, what about the others?
Just got it to work in the charts plugin:
Name: NOAA or something
Min/Max zoom, 1-24
Map source: WMTS
Format: png
URL: https://gis.charttools.noaa.gov/arcgis/rest/services/MarineChart_Services/NOAACharts/MapServer/WMTS
Add under Layers:
MarineChart_Services_NOAACharts
ok, so theoretically it is possible to setup the WMTS source as a proxied tilemap chart provider through SignalK with the modified charts plugin, but there is a catch: For some reason WMTS z value is offseted by 2 conpared to lets say openstreetmaps, e.g. for OSM if the zxy values are /12/x/y for the noaa charts it is /10/y/x, made a temporary change to my serve from cache function to handle url template like /{z-2}/{x}/{y} to subtract 2 from the z value and with that change it works. Here is a screenshot from my cached image directory
@Владимир Калачихин do you know why the values would be offsested like this?
what about the others?From the same file, you need to specify the parameters and projection in the url.
do you know why the values would be offsested like this?The matrix set doesn't have to be standard 😀 The TileMatrixSet default028mm same as the WorldMercatorWGS84Quad but 'Tile matrix Id'+2. So '/{TileMatrix}/{TileRow}/{TileCol}.png' === 'z+2/y/x.png' I do not know why this is so. My version of the GaladrielCache is currently somewhat broken in the process of a large update, so it's difficult for me to make a working example right now, sorry. But this: https://gis.charttools.noaa.gov/arcgis/rest/services/MarineChart_Services/NOAACharts/MapServer/WMTS/tile/1.0.0/MarineChart_Services_NOAACharts/default/default028mm/12/6161/4822.png is works.
@Ba2 and others: https://github.com/SignalK/charts-plugin/releases/tag/v3.2.0 published, with local caching & seeding support
GitHub
Release v3.2.0 · SignalK/charts-plugin
🚀 Features
Add support to cache & seed online tile maps, with optional custom http headers
PR: #41
🐛 Fixes
fix: v2 api provider startups
PR: #42
maybe a message on #announcements ? thought you might want to give the published version a go, just in case i botched somthing with my last minute tweaks, and then tell the world about it
Thank you @Teppo Kurki , as i mentioned in the PR, this was more effort that I initially thought, but hopefully it will be a useful addition to the charts-addin and be used by many members of the SignalK Community
@Day With the PR merged in now you can setup your NOAA WMTS source as tile layer here is my config for the NOAA maps taken from the charts.json file:
this will cache the tiles as you move around in FreeboardSK
Is there a way to see when tiles are cached?
they should appear under ~/.signalk/charts/noaa/
Or: how would you want to see that?
i mean the individual tiles should be stored there, however if you see the map in freeboard when the proxy option is set to true, everything you see should already be cached in the ~/.signalk/charts/noaa/ directory
Very nice addition to the plugin. Would you say, it fully replaces a mapproxy setup (which I use)
Hi Chris, I don't know the capabilities of your mapproxy setup but in theory it should be able to. It will replace mine for sure :).
The only thing i am missing with the current version is the ability to handle navionics tile maps.
True! I actually use no special features of mapproxy, I only do caching.
Sometimes mapproxy seems to be unreliable with an intermittent internet connection. So I guess I will give the new sk-proxy a try
Give it a go, feedback is appreciated 🙂
Is the charts directory in .signalk hard coded? The plugin seems to ignore my custom charts-dir path.
To be more specific: The plugin stores caches tiles in .signalk/charts, although I have configured a different directory as my charts-dir
I debugged through the code and the charts path configurations are used to load mbtiles and tile directories with correct tilemapresource.xml or metadata.json files in them and changing those directories wont effect the cache directory, so yes at the moment the cache dir is hard coded. I agree it would be nice to be able to configure that
Would indeed be nice. It would help to separate system from storage. At the moment I use another partition to store mbtiles and such (=charts-dir). Although the chart-plugin takes care of minimum disk space, I think it would be nice to also store the tiles-cache at a location that the user can specify.
created a new branch on my forked version of charts-addin and added support for configuring cache directory.
Lets wait for more feedback, if nothing serious comes up i will create a new PR for this
The only thing i am missing with the current version is the ability to handle navionics tile mapsLike this?
Or like this?
Or, may be, like this?
Yes pretty much so 😄
btw, how do you distribute those special maps? They are not part of the GaladrielCache code
behind the bar 😆
how do you distribute those special maps? They are not part of the GaladrielCache code😀 Making some curtsies to your world: https://github.com/VladimirKalachikhin/Galadriel-map/blob/master/emergencykit/GaladrielMap-2.21_GaladrielCache-2.10_gpsdPROXY-1_netAIS-2_e-inkDashboardModern-0_NoVectorTiles_.zip 😀
Works nicely, thank you
Hi,
Tried this new fantastic feature :), unfortunately with little luck...
Defined a region and tried starting a seeding job yesterday, screenshot.
No sign of a cache file or tiles in the charts directory..
Log is flowing with 304 messages, not sure how to interpret in this context (screenshot)
I think the third screenshot is when I started the job...
A few minutes later the process chrashed.... another screenshot
After the restart a cache job is started and is still running...more than 12 hours
Still writing these log entries as I write this...
Nov 28 08:56:28 dronten signalk-server[112732]: #033[0mGET /signalk/chart-tiles/cache/jobs #033[36m304#033[0m 2.861 ms - -#033[0m
No job was ever visible in the charts plugin UI..
Please let me know if I can assist with anything further.
/signalk/chart-tiles/cache/jobs are not jobs running but the seeding webpage repeatedly requesting "are there any jobs running" and getting an empty list, with code 304 Not Modified (= "no change since last time you asked")
if you share the map configuratio you are trying to seed (the region is less important, but helps also) then somebody(?) can try thisOK that is very logical with the 304:s maybe a bit ambitious to check and log every 60 seconds but I can see why 🙂
Map config is:
Url: https://map01.eniro.no/geowebcache/service/tms1.0.0/nautical2x/{z}/{x}/{-y}.png
Tilelayer
png
I tried with max zoom level 18 and this might be what in combination with the Region reaches the limit for my setup, see attached part of syslog.
With Max zoom level 15 (default) a job was created and is now running, see .
@Tore Dahl the 304 messages you see are the webapp for the signalk-charts trying to fetch the status of the jobs from the server, yes getting them every 2 seconds might be ambitious, but it is the way for the webapp to keep itself updated regarding the concurrent downloads going on in the background, once you close the webapp, they shouldn't flood your log anymore :). Also, when the proxy is enabled and you can see the map in freeboardsk, tiles that are fetched from remote should already appear in your configured cache directory by default it is ~/.signalk/charts. There should be directory named Eniro in your case filled with tiles.
Another thing to look at would be your remaining disk space, No caching will happen if your remaining disk space is less than 1GB.
Are you running this on the raspberrypi? if so how many gigs of RAM do you have?
I think i know where the issue is coming from
@Teppo Kurki Heap out of memory issue seems serious, do you think it is because of the memory on the RPi is not enough to keep track of all the await statements while downloading, if so I should have another look at the section of the code.
This is not on a RPi, it's a normal latop running Ubuntu, and memory is a bit more that usual RPi configs, see screenshot
I do have disk space as well and have FYI succesfully executed my way up to zoom level 17 for the region, worked out OK and used about 1,4 GB disk in the end.
See screenshot on completed seed jobs.
It seems to me that the number of tiles is to big causing an overflow, could that be the case?
I tried to get the entire Österlen region at 19 zoom level (2.5 million tiles), and i encountered the same issue, screenshot below. As I suspected nodejs didnt like creating 2.5 million await statements, but unlike my initial guess as least in my case i ran out of memory while checking which files are in cache before even starting downloading the tiles.
I will switch the implementation to use a thread pool, so we dont await 2.5 million function calls. At least this way the memory usage stays consistent, however the calculating all those tiles and then downloading them will take enormous amount of time. And the system becomes very laggy especially while calculating which tiles are inside the region, i wish there was a good way to stop the user from shooting themselves in the foot by creating big regions or asking for high zoom levels.
@Tore Dahl can you create an issue for this on github?
can you use something like a generator in python terms to generate / iterate the list of tiles dynamically? you don't want to precalculate the entire 2.5 million tiles.
you just need the bounding box and then an x/y iterator.
for example, with a given bounding box you should be able to calculate the upper left/lower right x and y indexes. that's just 4 integers you need to store.
I thought about it while writing the code, but i wanted to give feedback about the percentage of the work done, as seeding bigger jobs can take hours
you can still give feedback. if you now upper left / lower right x and y you can calculate percentages
do it as integer operations using the bounding box tile x/y coordinates.
max_x - min_x = tot_x
maxy min_y = tot_y
tot_tiles = tot_x * tot_y
cur_x - min_x = x_done
cur_y - min_y = y_done
x_done * y_done = tot_done
(100 / tot_tiles) * tot_done
(maths needs some work, i scratched that very quickly :-D)
but consider it all as integer operations on the total tile space, it will be much more efficient than storing e.g. every url you need to process. generate the urls as the last step.
for bounding boxes it would work, but regions can be convex shaped like L or U and if i calculate rough percentage over the bbox of the region, the results would be misleading
got it. what are you storing right now?
even for convex shapes, you could just store an array of x,y pairs?
i didn't do that maths on the storage costs of 2.5m x,y pairs, but if you serialized that to disk and didn't try and store it in ram it wouldn't be that bad.
or x,y,z triplets
or make the algorith that generates x,y,z tile labels from your convex shape into a generate so you can call back on it, like -> getNextTriple(): then add a getPercentDone() as the generator can store where it is is. instead of pre-calculating 2.5m.
Currently I store every every x,y,z triplet that intersects with the region or bbox, also prior to downloading the tiles I store every x,y,z triplet which is not cached. Storing these in memory is not the problem. Problem is to calculate which tiles intersect with the region as it takes time.
Thread pooling will fix the memory hog bug.
haha getPErcentDone requires knowing the total number of tiles 🤣
Lets see what @Teppo Kurki will say about this discussion besides "I told you to implement only boundbox seeding" 🙂
okay, i'm back in my box. whenever i hear things like "2.5 million await statements" my spidey sense goes off
Sounds like you are on the right track. You can offload the cpu intensive task to a worker to keep the main thread free
And then on to next todo: use mbtiles instead of the file system
Issue created:
https://github.com/SignalK/charts-plugin/issues/48
Sorry for stretching the limits on my first try 🙂
FYI, I have now incrementally seeded the region to level 15, 16 and 17 one at a time without breaking 🙂
So impressed and happy that this is working and will likely use the off-line capabilities extensivly next season!!
Thanks guys!!
Do not know whats behind the scenes, but when you create off-line copies of e.g. Garmin maps you are restricted to draw boxes, similar to bounding boxes but you can draw them.
They also do some calculation and there is a size limit on the boxes, likely to avoid similar problems.
Tried zoom level 18 with upto 17 already in the cache but fails anyway.
Question since the crash and following restart cleaned the seeding jobs, is there a way to see what is cached ?(other than filesystem)
@Tore Dahl You can re add the same job and it will tell you the total number of tiles and how many of them are cached, and give you new percentage of completion. It would be nice to see which tiles are coming from the cache and which once are freshly fetched in freeboardsk similar to SAS.Planet does, but at the moment i don't have a good idea on how to accomplish that. Perhaps by adding a new transparent layer on top
mbtiles would be a nice addition, it would also drastically improve disk usage and take less space.
I see that there is ongoing work switching to signalk/mbtiles library, perhaps we can wait for it to complete.
if we create an mbtiles file in the charts directory, it will be registered by both the offline map provider and the proxied online map provider
@night199uk A compromise solution would be to estimate the number of tiles which can be calculated very quickly based on some chatgpt implementation with around 5-10% accuracy and use the estimation as a base to calculate progress, while the real seeding can be done using a generator function. minimal memory usage while giving useful feedback to the user
how does the function that generates the list of tiles for a given shape file work? i assume you are reading some complex shape file, e.g. geojson or gpx or some such and then generate a list of tiles from it?
i should probably go look at the implementation before i comment 😄
i hate estimated functions if i can possibly avoid them
https://github.com/SignalK/charts-plugin/blob/2c8e3a777e412d466b6bdbeb2c6a6239b1a569b1/src/chartDownloader.ts#L393
This is how the tiles are calculated now, proposed implementation converts this to a generator
Based on a few tests i did estimation is 100-300 times faster than calculating the real set of tiles and the results are within 3-5%
Also if people put up with microsofts copy file progress bar decades, i am sure this will be sufficient for the use case 😄
@chris I did similar in python with supermercado library, but as the number of tiles increase memory usage increases and possibly calculation time increases as well (not tested it for this case).
Both memory and computation are scarce resources on a raspberry pi
What I did in a script is to first run a loop, in which I determine the tiles coordinates that I need to cache and count them. This is actually rather quick, eben vor 100s of thousands of tiles. Then afterwards a loop to actually download the tiles (and show progress compared to the pre-determined number of tiles). I did not found this to be too memory intensive.
suggestion from me, instead of this:
const tiles: Tile[] = []
tiles.push({ x, y, z })
I would generate the list of tiles to be cached directly to disk. tile downloading is a very long process and this allows you to restart the server and continue downloading.
by writing this to disk as well you also avoid the memory hit of keeping the list of tiles in memory.
depending how you format the file, calculating the number of tiles referenced in a file could be very fast. e.g. if you store them as sets of 3 fixed 64-bit integers, you just need to divide the size of the file by 24 bytes (3*8 bytes); and you know how many you have. there is probably a fast way to count even if store them as text.
then your consumer just needs to read the file line-by-line. great thing is caching this many tiles could take a long time so doing it like this allows you to restart the server and continue caching.
and no memory consumed for enormous tile lists; use disk space instead (which hopefully is plentiful)
perhaps have a 'low disk space check' for both this file and the tiles themselves so we don't allow the tile downloader to fill the disk. just in case.
thinking about it more, i would serialize the tile-refs to disk and just keep a count of the tile-refs in memory. as you iterate the tile-refs on disk decrement the count.
if the server restarts and you lose the count, you can do a semi-expensive operation to count the number of tiles in the file when the server starts (worst case spawning 'wc -l' is ludicrously fast, but there are better ways probably).
lol, I knew that this feature is not quite as simple as you'd think 😆
the plugin already has a guard against low disk space
the mbtiles work is really dependent on switching to a newer Node version. one way to move forward is that this feature can write to mbtiles directly with node:sqlite, not creating more dependency on the old mbtiles library, and make that feature available only when node:sqlite is there
and yes, coexistence with the existing mbtiles feature needs to be addressed, but certainly doable