Skip to content

Commit c42daa7

Browse files
author
Graham Palmer
committed
iox-#2128 Draft design for named segments
1 parent 154b5a5 commit c42daa7

File tree

1 file changed

+228
-0
lines changed

1 file changed

+228
-0
lines changed

‎doc/design/draft/named-segments.md‎

Lines changed: 228 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,228 @@
1+
# Named Segments
2+
3+
## Summary and problem description
4+
5+
Shared memory configuration, as handled by the RouDi config and described in detail in
6+
[the configuration guide](../../website/advanced/configuration-guide.md) currently works by
7+
declaring one or more **segments** backed each by one or more **memory pools**.
8+
9+
Segments currently support configuring access controls, allowing one to specify which processes are allowed to read from/write to them.
10+
These access controls are also used to determine which segment a publisher may use to publish messages, and
11+
from which segments a subscriber is allowed to receive messages. This coupling presents a number of implications:
12+
13+
* The name of shared memory segments created under `/dev/shm` is automatically deduced as the name of the
14+
POSIX user group that has write access to it.
15+
* The above means that there can only be one segment that any given group has write access to.
16+
* In order to determine which segment a publisher shall publish to, the permissions of the containing process are
17+
matched against the permissions of each segment to identify a *unique match*.
18+
* It is therefore not possible to have a single publisher access multiple segments.
19+
* It is also not possible to have multiple publishers in the same process which access different segments
20+
21+
Because of the last point, it is impractical to split topics across several segments - doing so requires publishers
22+
of different topics to be run in processes under different users with different group membership. Instead, applications
23+
must rely on configuring differently sized memory pools to support topic data of different sizes.
24+
This has a number of drawbacks, including:
25+
26+
* Processes communicating over otherwise independent topics share the same memory pool.
27+
If any one process loans too many chunks, or fails to release chunks, it can cause starvation of ALL processes
28+
relying on this memory pool.
29+
* Topic data allocation is assigned to memory pools via "bucketing" - or finding the memory pool with the smallest
30+
chunk size to support the loan request. This means that even if one attempts to cater a memory pool to a specific topic,
31+
any other topics with the same size requirement will be forced to use the same memory pool.
32+
* Faulty publishers which e.g. write past the end of their loaned message may cause faults in any process communicating
33+
over the same shared memory segment. These faults will be difficult to trace, especially because the faulty publisher
34+
may never write past the end of the mapped shared memory segment, and therefore may never trigger a segfault.
35+
36+
The above points demonstrate how the current design violates the principal of [Freedom From Interference](https://heicon-ulm.de/en/iso26262-freedom-from-interference-what-is-that/).
37+
38+
### Proposal
39+
40+
In order to overcome the above limitations, shared memory segments shall be mapped by
41+
name instead of by access control. This will increase flexibility while still granting all the benefits of the current access control model.
42+
43+
## Requirements
44+
45+
* The RouDi configuration must support specifying names for shared memory segments.
46+
* If a name is not configured, the fallback behavior must be the same as it is currently - assigning the name of the current process user.
47+
* Communication endpoints should be able to specify which segments they intend to use.
48+
* Publishers may specify a single segment into which they will publish.
49+
* Subscribers may specify one or more segments from which they may receive messages.
50+
* Clients and Servers may specify a single segment into which they send requests/responses, and multiple segments from which they receive responses/requests.
51+
* When no segment is specified, the fallback behavior must be the same as it is currently
52+
* Publishers will select the *unique* segment they have write access to.
53+
* Subscribers will receive messages from all segments they have read access to.
54+
* Clients and Servers follow the same rules as Publishers for data they send and the same rules as Subscribers for data they receive.
55+
* Creation will fail if a communication endpoint requests to use a segment to which it does not
56+
have the proper access rights.
57+
58+
### Optional additional requirements
59+
60+
* When initializing the runtime, a list of segment names may be provided. In this case, rather than mapping all shared memory segments available in the system, only those that are named shall be mapped.
61+
* When a communication endpoint requests a segment that has not been mapped, this may be configured to either result in a failure or in mapping the requested segment on the fly.
62+
63+
## Design
64+
65+
### Updated RouDi Config
66+
67+
Supporting named segments will require add an additional `name` field to the RouDi config under the `[[segment]]` heading, as well as updating the config version Example:
68+
69+
```
70+
[general]
71+
version = 2
72+
73+
[[segment]]
74+
name = "foo"
75+
76+
[[segment.mempool]]
77+
size = 32
78+
count = 10000
79+
```
80+
81+
Updating the version will be necessary for older versions of RouDi to fail informatively when presented with a newer config. Otherwise RouDi can support version 1 configs as if they are version 2 configs with no `name` fields specified.
82+
83+
### Requesting Segments
84+
85+
#### When creating a Publisher
86+
87+
A new field will be added to the [PublisherOptions struct](../../../iceoryx_posh/include/iceoryx_posh/popo/publisher_options.hpp) as follows:
88+
89+
```
90+
struct PublisherOptions
91+
{
92+
...
93+
ShmName_t segmentName{""};
94+
```
95+
96+
#### When creating a Subscriber
97+
98+
A similar field will be added to the [SubscriberOptions struct](../../../iceoryx_posh/include/iceoryx_posh/popo/subscriber_options.hpp), except that it will support multiple elements
99+
100+
```
101+
struct SubscriberOptions
102+
{
103+
...
104+
vector<ShmName_t, MAX_SUBSCRIBER_SEGMENTS> segmentNames{};
105+
```
106+
107+
`MAX_SUBSCRIBER_SEGMENTS` can be set to some reasonably small value to start with since explicitly allowing many but not all read-access segments is an unlikely use case.
108+
109+
If this assumption turns out to be wrong however, we can always update it to be `MAX_SHM_SEGMENTS` instead.
110+
111+
#### When creating Clients and Servers
112+
113+
Clients and Servers will have similar fields, distinguished by request/response:
114+
115+
```
116+
struct ClientOptions
117+
{
118+
...
119+
ShmName_t requestSegmentName{""};
120+
vector<ShmName_t, MAX_RESPONSE_SEGMENTS> responseSegmentNames{};
121+
```
122+
123+
```
124+
struct ServerOptions
125+
{
126+
...
127+
vector<ShmName_t, MAX_REQUEST_SEGMENTS> requestSegmentNames{};
128+
ShmName_t responseSegmentName{""};
129+
```
130+
131+
#### When initializing the runtime
132+
133+
We may wish to also additionally support requesting segments upon runtime initialization. The motivation here is that if we know we will only need to use certain segments, we can avoid expending unecessary resources mapping the ones we will not use. Additionally, this can be used as an error mitigation technique to prevent communication endpoints from requesting an unsupported segment.
134+
135+
Currently initialization of the runtime happens in the constructor, and any errors that occur result in a contract violation and program termination.
136+
137+
While it is not directly in scope of this design, it would be beneficial to refactor the Runtime implementations to use the builder pattern such that any errors may be handled through returning errors types.
138+
139+
Tying this together with requesting segments, this could look something like:
140+
```
141+
class DefaultRuntimeBuilder
142+
{
143+
...
144+
IOX_BUILDER_PARAMETER(vector<ShmName_t>, shmSegmentNames, {});
145+
IOX_BUILDER_PARAMETER(bool, allowUnmappedSegments, false);
146+
...
147+
public:
148+
expected<DefaultRuntime, RuntimeBuilderError> create() noexcept;
149+
```
150+
151+
The `create` method of this builder would then return a custom error if a
152+
segment was requested that either does not exist or the current process does not have access to.
153+
154+
Additionally, when a publisher or subscriber is created, the `allowUnmappedSegments` would determine whether or not we fail or try to map the newly requested segment on the fly.
155+
156+
### Segment Matching Algorithm
157+
158+
Ensuring backwards compatibility means carefully crafting how we register segments. The following pseudocode demonstrates how this should work in different scenarios
159+
160+
#### Endpoints requesting write access (Publisher, Client Request, Server Response)
161+
162+
In RouDi, given:
163+
* `userGroups` - A list of POSIX user groups a publishing process belongs to called
164+
* `segmentName` - The (possibly empty) name of a shared memory segment specified in the publisher options call
165+
* `mappedSegments` - A list of shared memory segments
166+
167+
1. If `segmentName` is not empty
168+
1. Find the segment in `mappedSegments` matching the `segmentName`
169+
2. If the containing process has write access to the segment via one of its `userGroups`, return the corresponding segment information
170+
3. Otherwise return an error indicating the publisher does not have write access to the requested segment
171+
2. If it is empty:
172+
1. Iterate over all process `userGroups`
173+
2. Determine if the user group name matches the name of one of the segments
174+
3. If it does:
175+
1. If a match has already been previously made, return an error indicating that the publisher must only have write access to one segment
176+
2. If not, verify the process has access to the segment and record the match
177+
4. At the end of iteration, if a matching segment has been found, return the segment information
178+
5. Otherwise return an error indicating that no matching segment has been found
179+
180+
#### Endpoints requesting read access (Subscriber, Client Response, Server Request)
181+
182+
In RouDi, when handling a port request for a new subscriber (and WLOG for clients and servers), given:
183+
* PublisherOptions for a single publisher publishing on the requested topic
184+
* SubscriberOptions for the requested subscriber
185+
186+
We determine if the endpoints are compatible as follows (and we repeat this for each publisher if there are several):
187+
1. To determine if the subscriber and publisher segments match:
188+
1. If the list of segment names provided by the subscriber is non-empty, check if any name matches the name provided by the publisher (an empty name is a non-match)
189+
2. If the list of segment names provided by the subscriber is empty, then it is always a match
190+
2. Determine if blocking policies are compatible
191+
3. Determine if history request is compatible
192+
4. Return true if all cases are met
193+
194+
If no publisher is compatible with a subscriber, then RouDi will refuse to provide a port, as is the current behavior when subscribers have incompatible blocking policies.
195+
196+
#### Runtime requesting segments to map
197+
198+
In the client process, while initializing the runtime, given:
199+
* `userGroups` - A list of POSIX user groups a publishing process belongs to called
200+
* `segmentContainer` - A list of shared memory segment information
201+
* `segmentFilter` - A (possibly empty) list of segment names to filter against
202+
203+
In order to determine which segments to map:
204+
205+
1. If `segmentFilter` is empty then
206+
1. Determine which segments the process has access rights (read or write) to.
207+
2. Map all of them.
208+
2. If `segmentFilter` is not empty
209+
1. Iterate over each name in the filter
210+
2. If there is a segment that matches the name in the filter
211+
1. If the process has access rights to that segment, map it
212+
2. If not, return an error indicating that a segment was requested the runtime does not have access to.
213+
3. If there is no segment that matches the name in the filter, return an error indicating that there is no segment matching the one requested to be mapped.
214+
215+
## Development Roadmap
216+
217+
- [ ] Extend the RouDi config to support specifying segment names
218+
- [ ] Add the name to the `MePooSegment` data structure and populate it based on the RouDi config
219+
- [ ] Update communication endpoint options structs to include segment names
220+
- [ ] Update the publisher segment selection logic to take the segment name into account
221+
- [ ] Update the subscriber compatibility check to check for a segment name match
222+
223+
### Optional
224+
225+
- [ ] Refactor runtime initialization to use builder pattern
226+
- [ ] Add segment filter and apply it to segment mapping during runtime initialization.
227+
- [ ] Add flag to specify whether endpoints requesting non-mapped segments should fail or whether segments should be created dynamically
228+

0 commit comments

Comments
 (0)