Using Shared Memory with OSRM

If you have multiple applications accessing OSRM data then it does not make sense for each of those to have a separate copy of the data resident in memory. This is especially true if you’re using a relatively large map, in which case memory consumed by multiple processes might be enormous.

An alternative is to store the map data in shared memory, allowing multiple processes to access a single copy of the data. The official OSRM documentation for using shared memory can be found here. This post gives further details.

Adjust Limits

You need to configure your system first. There are both system-wide and user-specific changes.

System

Check on the default shared memory limits.

$ ipcs -lm

------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 18014398509465599
max total shared memory (kbytes) = 18014398509481980
min seg size (bytes) = 1

Those numers are pretty enormous and will probably be sufficient. But if you are using a really big map then you might need to expand them. You can do this by setting the shmall and shmmax kernel parameters. We’ll go the other way and actually reduce the allocations. Add something like the following to the end of /etc/sysctl.conf:

kernel.shmall = 1000000000
kernel.shmmax = 10240000000

The shmmax parameter specifies the maximum amount of shared memory (in kB) that can be requested by a single process. The shmall parameter sets the total amount of shared memory pages that can be used across the entire system. A page is simply a block of memory. You can check on the page size (in bytes) as follows:

$ getconf PAGE_SIZE
4096

Now reload the system settings.

sudo sysctl -p

Use ipcs again to check that the changes have been applied.

$ ipcs -lm

------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 10000000
max total shared memory (kbytes) = 4000000000
min seg size (bytes) = 1

User

There are separate limits which apply to individual users.

$ ulimit -a | grep max
max locked memory       (kbytes, -l) 16384
max memory size         (kbytes, -m) unlimited
max user processes              (-u) 7729

To share a map of any reasonable size we’d need to boost those. Add the following to /etc/security/limits.conf (the settings will be applied to the ubuntu user).

ubuntu           hard    memlock         unlimited
# Memory is in units of kB.
ubuntu           soft    memlock         8388608

Now logout and login again so that the changes take effect. Check.

$ ulimit -a | grep max
max locked memory       (kbytes, -l) 8388608
max memory size         (kbytes, -m) unlimited
max user processes              (-u) 7729

Using Shared Memory

Now we are ready to add OSRM data to shared memory. Suppose that we have data which has already been prepared for use with OSRM.

$ osrm-datastore durban.xml.osrm

Check.

$ osrm-datastore --list
[info] name       shm key timestamp       size
[info] /static          0         1       16519676
[info] /updatable       1         1       34074239

Now you can launch the OSRM router using data from shared memory.

$ osrm-routed --shared-memory

It’s also possible to have multiple datasets in memory simultaneously. First let’s clean up (this will remove the existing OSRM data in shared memory).

$ osrm-datastore --remove-locks
$ osrm-datastore --spring-clean

Now add two maps, giving each a specific name.

$ osrm-datastore --dataset-name durban durban.xml.osrm
$ osrm-datastore --dataset-name pietermaritzburg pietermaritzburg.xml.osrm

Check.

$ osrm-datastore --list
[info] name                        shm key timestamp       size
[info] durban/static                     0         1       16519676
[info] durban/updatable                  1         1       34074239
[info] pietermaritzburg/static           2         1       5691727
[info] pietermaritzburg/updatable        3         1       9880003

You can now launch the OSRM router using either of these shared datasets.

osrm-routed --shared-memory --dataset-name=pietermaritzburg

Notes

  • According to this thread it’s only possible for OSRM to create shared memory objects for one user. So, for instance, you cannot run osrm-datastore for one map as root and then again for another map as another user. This will result in a “Permission denied” error.
  • If you get a “Permission denied” error when running osrm-datastore it might help to reboot the machine.
OSRM Linux
Avatar
Andrew B. Collier
Entrepreneur / Data Scientist

Related