Library Design a Feature Flag Service AgileViper46
Design a Feature Flag Service — system design by AgileViper46 Lean Hire Reviewed by 6 specialized AI reviewers. Explore the diagram and the full per-section feedback below.
Author's explanation How the author described their design We have a feature flag service which would be handling the crud operations on the feature flags This service will be solely responsible for creating modifying deleting or updating the feature flag the feature flag service will be using transactions to make sure whenever an update is confirmed we store that update into the audit table as well as we update it into an outbox event which will be used later in the system Once this is done YCDC the rules get updated into an hcd cluster so hcd stores the feature flag name or a feature flag id as a key and the value is the rule now every applications via the sdk can connect to this hd cluster via grpc we are using grpc because we need extreme low latency and grpc will provide it and itcd supports grpc connexions so via the sdk we can directly connect and get the value of that key when we want to evaluate it Now the evaluation logic for a feature flag would lie in the client's hand The client is going to evaluate the feature flag for us The sdk will The sdk will get the rule and then it will evaluate that particular rule across this particular user so consider this case where we have targeted users then we just extract the user from the jwt we validate whether that user exists in the feature list or not if it exists then we pass it if its lets say by region then we evaluate by the region if its percentage of the users then what we can do is we can hash that user based on certain hashing algorithm and make sure that it lies into the first 30% or 20% of the hash range then we enable it else we disable it Now let's come to some numbers Ideally the read and write operations for the feature flag are not going to be that much. Considering that we're creating updating everything would be less than 10,000 requests per second because we have more evaluations which will be done by the applications but the updation or the creation of a feature flag is ideally not that much so we would be able to easily handle via cd Google Postgres Sql Consider that we have let's say 10 million feature flags Each feature flag would be taking approx 1 hundred and fifty bytes That's just 1.5 gb of data and considering the audit trail where we have for an average of ten rows per feature flag that will become around 15GB of data so it can be easily handled by a single post press instance Consider a skilling of let's say 1 million evaluations per second and it city clusters can do an evaluation one instance of an xcd can easily handle approximately 50,000 requests per second That means we can easily do one million via 20 clusters To speed up the local evaluation our sdk can also perform caching on the rules instead of getting the rule every time from the hct What it can do is we can cash it locally and then evaluate against the rule for a certain time and maybe then we can you know go to that city cluster after sometime the dtls pass to fetch the new rule To make this even more robust and make sure that the feature flag effects instantaneously we can also provide the sdk can do a watch notification on the key in that city cluster so whenever the rule changes that city is going to notify our sdk and sdk is going to update the key update the rule basically
AI
Design Review Hire Signal Lean Hire
This is a credible senior-level design with several strong architectural choices, especially the control-plane/data-plane split, client-side evaluation, and outbox-based propagation. The main concerns are not that the design is wrong, but that it is under-specified in several important areas: failure semantics, consistency expectations, entity modeling, and API rigor. That keeps it below a clear 'hire', but it is still materially stronger than a mediocre answer.
AI Review Senior (L5-L6) ⭐ Excellent Latency target is tied to client-side evaluation
The design explicitly uses SDK-side rule evaluation with local caching/watch-based refresh, which is a strong NFR-driven choice for the stated <10ms evaluation goal. This avoids putting the full evaluation path on a remote service for every request and shows awareness that network hops would dominate latency at 1M evaluations/sec.
✅ Good Availability and auditability influence the write path
The explanation connects feature flag updates to transactional persistence of both the flag change and an audit record, plus an outbox event for propagation. That is a good sign that audit history and rollback are being treated as first-class non-functional requirements rather than afterthoughts.
✅ Good Scale numbers are at least connected to the stated assumption
The candidate does not leave the 1M evaluations/sec assumption floating in isolation; they estimate write traffic separately, size stored flag/audit data, and reason about read scaling for the rule distribution layer. The exact numbers may be rough, but the methodology is relevant to the stated workload.
warning Availability is stated but not translated into a concrete failure model
You say the system needs to be highly available, but what happens when the rule store or watch channel is unavailable or partitioned? With SDK caching, clients may continue serving stale rules indefinitely; without caching, evaluations could fail outright. You should define the expected behavior under control-plane failures: fail-open vs fail-closed, maximum tolerated staleness, and how long SDKs can safely operate from cache.
warning Consistency model for flag propagation is implicit, not justified
Have you considered what consistency guarantee teams expect after toggling a flag? The design appears eventually consistent because updates flow through storage and watches into SDK caches, but the NFRs include rollback support and dynamic enable/disable. If rollback must take effect quickly, stale SDK caches can violate that expectation. You should explicitly state the consistency model for reads after writes and explain what breaks if propagation is delayed.
warning The <10ms target is not clearly scoped
What exactly is the latency SLO measuring: local SDK evaluation only, cache miss fetch plus evaluation, or end-to-end time from a flag toggle to all clients seeing the new value? The current design can likely meet <10ms for local evaluation, but not necessarily for cold starts or propagation after updates. Narrowing the metric would make the target defensible and prevent hidden gaps.
info Rollback support needs an explicit timeliness objective
You could improve this by defining an operational rollback SLO, for example how quickly a bad flag change must be reversible across clients. Right now audit history is covered, but rollback as an NFR is only partially addressed unless you specify how fast prior versions can be restored and propagated.
🗃️ Core Entities Review
✅ Good FeatureFlag and audit history are identified separately
The candidate distinguishes the mutable feature flag definition from its audit/history records, which is a sensible domain split for a system that must support create/update/delete plus traceability of changes.
warning Core actor/entity modeling is muddled
Have you considered what the primary ownership model is for flags? The requirements say developers create flags and teams enable/disable them, but the entities only list 'Application(user)' without clearly separating Application, Developer/User, and Team. Without a clear actor model, it becomes hard to express who owns a flag, who can change it, and which team is toggling it.
warning Targeting concepts are embedded but not modeled as entities or relationships
What happens when a flag needs multiple rollout rules at once—such as targeted users plus region plus percentage rollout? The explanation talks about rules, targeted users, and regions, but the entity list does not model a Rule/Targeting entity or its relationship to FeatureFlag. That leaves the core happy path under-specified because the system revolves around storing and evaluating these targeting definitions.
warning Relationships between entities are not defined
Have you considered how these entities connect? For example: does one Team own many FeatureFlags, can one FeatureFlag have many targeting rules, and does one FeatureFlag have many History records? Those relationships are central to the CRUD and evaluation flow, but they are not made explicit, which makes the domain model feel incomplete for a senior-level design.
✅ Good Ties the 1M eval/sec assumption to a concrete serving estimate
The candidate does at least translate the stated scale target into infrastructure sizing by estimating per-node etcd throughput and deriving a rough cluster count. The exact number may be debatable, but the methodology of starting from evaluation QPS and mapping it to serving capacity is directionally correct.
✅ Good Separates control-plane writes from evaluation traffic
They explicitly distinguish low-rate CRUD/update traffic from high-rate flag evaluation traffic. That is an important capacity insight for feature flags because it avoids over-sizing the write path based on read-heavy assumptions.
✅ Good Uses local SDK caching/watch to reduce steady-state read load
For this workload, pushing rules to SDK caches and using watches for updates is a sensible capacity lever. It shows awareness that the raw 1M eval/sec should ideally not all hit the backing store continuously.
warning Peak evaluation path is internally inconsistent
Have you considered what happens if SDK cache hit rate is low, clients restart, or many instances reconnect at once? The design first sizes etcd for the full 1M eval/sec, then says SDKs cache rules and watch for changes, which implies the backing store should mostly serve refreshes rather than every evaluation. You should make one clear capacity model: steady-state cache-hit traffic, cache-miss traffic, and reconnect storms.
warning Write-path estimate is asserted, not derived
Have you considered how many teams, flags, and update frequency would actually produce the claimed '<10,000 requests/sec' on create/update/delete? For senior-level capacity planning, I would expect a short derivation from tenants/teams and average flag churn, because the control plane also drives fanout, watches, audit growth, and replication load.
warning Storage sizing ignores important growth dimensions
Have you considered what happens when rules become more complex, targeted user lists grow large, or audit retention extends over time? The estimate of 150 bytes per flag is likely only metadata-scale and does not clearly include targeting lists, region rules, percentage rollout config, indexes, row overhead, or replication. The current storage math is too optimistic to justify 'single Postgres instance' confidently.
warning No bandwidth or connection-capacity model for SDK watches
Have you considered what happens when a large fleet of application instances maintains long-lived watch connections and a popular flag changes? Even if evaluation is local, the system still needs capacity for concurrent gRPC connections, watch fanout, and update bursts. A senior-level answer should at least acknowledge connection count and change-propagation bandwidth.
info Include replication and failure headroom in node sizing
You could improve this by asking: if one etcd node or one database replica is unavailable, does the remaining capacity still meet the assumed load? Capacity numbers are more convincing when they include some failover margin rather than sizing exactly to nominal throughput.
✅ Good Core management APIs are mostly covered
The routes support the main admin flow in the requirements: create, fetch, update, toggle, and delete a feature flag. The separate PATCH for enable/disable is a reasonable API choice for a very common operation.
✅ Good Protocol split matches the usage pattern
Using REST-like endpoints for low-frequency flag management and gRPC for high-frequency evaluation is a sensible separation. Given the stated 1M evaluations/sec assumption, calling out gRPC for the hot path shows awareness that the read/evaluation path has different latency needs than CRUD.
warning Evaluation API is underspecified for real client use
What happens when a client needs to evaluate by flag name, environment, or a user context with attributes like region and user ID? `eval(featureFlagId, user) -> true | false` is too thin to be safely interoperable because the required request shape is unclear. You could improve this by defining the evaluation request explicitly, e.g. flag identifier plus a typed evaluation context containing userId and region, so SDKs and services evaluate consistently.
warning Rule modeled as a free-form string is brittle
Have you considered what happens when different clients parse `rule: "users : [*] | users : [30%] | users : [canada,brazil]"` differently or when validation is inconsistent? A string-encoded DSL makes the API hard to validate, version, and evolve. The client could send an invalid or ambiguous rule and the system may accept it but evaluate it inconsistently. A structured rule schema would make the contract much safer.
warning No clear error contract or retry semantics
What does the client see when a create fails validation, a flag is not found, or an update conflicts with a concurrent change? The routes list success shapes but not status codes, error bodies, or whether retries are safe. Without that, SDKs and callers cannot distinguish bad input from transient failures. You could improve this by documenting basic 4xx/5xx behavior and whether POST/PUT/PATCH operations are idempotent or require version checks.
info List/query APIs may become necessary for operability
There is no list endpoint, so teams can only fetch by ID. If the intended usage is strictly direct lookup, that is acceptable for the stated requirements, but in practice teams usually need to browse or filter flags. If you add that later, make sure it includes pagination and filtering rather than returning unbounded result sets.
info Resource naming could be cleaner
The CRUD routes are understandable, but the API would be easier to evolve if the rule and targeting concepts were represented more explicitly in the resource model instead of overloading a single `rule` field. For example, separating `enabled`, `targeting`, and rollout configuration would make partial updates and validation clearer.
⭐ Excellent Read path is separated from write path
Using Postgres for authoritative CRUD and audit history, then propagating changes to etcd for serving evaluations, is a strong architectural split for this problem. It keeps the low-QPS management workflow away from the high-QPS evaluation path and aligns well with the stated 1M eval/sec assumption.
⭐ Excellent Outbox/CDC propagation shows good failure-aware thinking
The candidate explicitly describes writing the feature flag change, audit row, and outbox event transactionally before publishing to etcd. That is a strong choice because it avoids losing updates between the database write and cache/config-store propagation, which is a common production failure mode.
✅ Good Client-side evaluation reduces central bottlenecks
Pushing rule evaluation into the SDK is a good design decision for the <10ms requirement. It avoids sending every evaluation through the feature flag service and makes targeted-user, region, and percentage rollout checks cheap at the edge of the application.
✅ Good SDK cache plus watch-based refresh is a sensible latency optimization
The explanation goes beyond just saying 'cache it' and proposes local SDK caching with etcd watch notifications to refresh rules. That is a practical way to reduce repeated remote reads while still keeping flag changes reasonably fresh.
critical End-to-end evaluation flow is not fully connected on the diagram
What happens when the application wants to evaluate a flag? The diagram shows Application and SDK as separate nodes, but there is no connection between them, and there is no drawn path from Application to the evaluation result. As drawn, the request flow does not complete end-to-end for the core read path. You should explicitly show Application -> SDK and clarify whether the SDK is an in-process library, sidecar, or remote service.
warning etcd becomes the first scaling and availability risk
Have you considered what happens if many SDKs across many applications all read or watch etcd directly at the stated scale? Even with local caching, a large fleet reconnect storm, deployment wave, or etcd leader change can create a thundering herd against the cluster. The design would be stronger if you explained how clients are partitioned, how watches are managed at scale, or whether there is an intermediate distribution layer instead of every SDK talking to etcd.
warning Single Postgres instance is a write-path SPOF
What happens when the Postgres node is unavailable? Developers and teams would lose the ability to create, update, delete, toggle, and record audit history, and new flag states would stop propagating to etcd. Since high availability is a stated requirement, the design should call out replication/failover for the source of truth and outbox processing path.
warning Staleness and rollback behavior on SDK caches is underspecified
Have you considered what happens if an SDK misses a watch event, is partitioned from etcd, or keeps serving a stale cached rule after a rollback or emergency disable? In this design, local caching is great for latency, but without a clear versioning/revalidation strategy, applications may continue evaluating with old rules. You could improve this by defining cache invalidation semantics, version numbers, and fallback behavior when watches break.
info Some components are underexplained in the HLD
You could improve this by making the role of Teams, Developer, Application, and SDK more explicit in the architecture. Right now Teams and Developer are just clients of the gateway, while Application is effectively orphaned in the drawn flow. A clearer separation between control plane (CRUD/toggles) and data plane (evaluation) would make the design easier to reason about under failure.
Want this kind of feedback on your own design? Draw your architecture for Design a Feature Flag Service and get an instant hire/no-hire signal from 6 specialized AI reviewers — free to start.
DrawLint.ai