Observe
Test runs
A single result does not explain whether a test is actually unstable.
Menu
I redesigned an advanced test-reliability workflow around signals, scoring, thresholds, and actions. This made complex automation easier to understand and trust without reducing control.
Automated tests do not always behave consistently.
The same test can pass in one run and fail in another, leaving engineering teams to determine whether they are seeing a meaningful issue or simply unreliable test behavior.
A developer-tooling platform was expanding its reliability detection system to support more configurable logic: which signals mattered, how strongly they contributed, when instability crossed a threshold, and what should happen afterward.
The capability was powerful. The configuration experience made that power difficult to reason about.
My role was to turn a dense set of technical parameters into a workflow teams could understand and configure with confidence.
Reliability logic
Same test.
Different outcomes.
Observe
A single result does not explain whether a test is actually unstable.
Interpret
Decide
Design problem
Flexible system.
Weak mental model.
Different engineering teams have different definitions of an unreliable test.
Some care primarily about repeated failures. Others care about timing variability. Some need issues surfaced quickly; others want stronger evidence before creating follow-up work.
That flexibility required significant configuration.
The early experience exposed most of that configuration at once. Each control made sense independently, but the interface left users to infer how signals, weights, monitoring periods, thresholds, and actions worked together.
The core problem was not the number of settings.
It was that the relationship between them was invisible.
The missing relationship
Visible settings. Invisible logic.
A conceptual view of the configuration burden.
Combining evidence
How do these signals combine?
Interpreting the result
Which settings change detection?
Triggering follow-up
When does the system actually act?
The controls were explicit.
Their relationships were left to the user.
Instead of organizing the experience around the parameters the system exposed, I reframed it around the decisions the user was making.
The reliability model
Four connected decisions.
One legible system.
Signals
Choose the signals the system should evaluate.
Weighted score
Define weights and observation periods.
Threshold
Set the threshold at which instability becomes significant.
Action
Configure downstream behavior.
System behavior follows the evidence.
Making the path from evidence to action explicit gave the configuration a legible mental model.
Illustrative signals, values, and response.
The first concepts treated advanced detection as another large section inside an already complex page.
Instead, I separated the decision to use an advanced detection model from the work of configuring it.
The default experience stayed understandable at a glance. Teams that needed more control could deliberately enter the advanced workflow.
Choosing the detection model
Reconstructed sequence
Default → choice → configuration
Default experience
Detection model
Use the system’s built-in model to identify unreliable tests.
Advanced settings stay outside the default experience.
Intentional entry
Change detection type
Choose how unreliable tests are identified.
Use the built-in detection model.
Define your own signals, thresholds, and actions.
Connect an external detection source.
Choose the model first. Configure its decisions next.
Progressive disclosure
More control appears when the user chooses it.
Advanced configuration became an intentional mode instead of additional complexity added to the default experience.
Reconstructed interface. Illustrative labels and selection state.
Reliability configuration can affect how teams interpret test results and when automated work gets created.
Adopting new detection logic meant changing a system teams already relied on. The experience needed to give them a way to evaluate that change.
Advanced detection could run alongside the current system, giving teams time to configure it and observe its behavior before deciding to switch.
The interface made the trial state explicit with Running in parallel and kept activation a separate decision.
Trying advanced detection
The current system stays active throughout the trial. The team decides when to make the change.
Confirm what will run and what stays in place.
Confirmation
Run alongside the current system?
Current detection stays active.
Start trialObserve the new behavior before relying on it.
Remains in use during the trial.
Runs for evaluation.
Starting a trial does not switch the active system.
Make advanced detection active through an explicit choice.
A separate decision after evaluation.
Teams could evaluate advanced detection before choosing to rely on it.
Static interface reconstruction. Labels and workflow states are illustrative.
Once advanced detection was configured, the main page did not need to reproduce every field.
Instead, I used it to summarize the active model. Weighted signals appeared as a compact table. Thresholds and actions became a separate configuration group. Detailed editing moved one level deeper.
01 Show me what the system is doing.
02 Give me a clear way to change it.
It no longer had to function as both overview and giant form.
Detection configuration
Evidence model
Two signals currently contribute to the score.
Evaluation window
Decision rule
Classify the test as flaky.
Create follow-up work automatically.
The parent view summarizes the active model while keeping detailed configuration one level deeper.
Static interface reconstruction. Values and labels are illustrative.
Advanced reliability logic still required technical configuration. The solution was not to remove that complexity. It was to reveal it at the point where it became relevant.
Weighted scores
Teams could define how strongly each signal contributed to the overall score and how long the system should observe it. Repeated percentage, duration, and unit controls moved out of the parent page and into one focused task.
Thresholds and actions
The threshold answered when the evidence became meaningful. The action answered what the system should do when that happened. Keeping them together made the relationship visible without treating them as the same decision.
Downstream work
Actions such as filing a bug introduced another layer of configuration: template, priority, and metadata. Those options appeared only when that action was selected.
Focused configuration
Each layer stays focused on one question, while deeper configuration appears only when the preceding choice requires it.
Evidence
Define how strongly each signal contributes and how long the system should observe it.
Signal model
One focused task
Decision
Define when the evidence matters, then decide what the system should do.
Threshold
The combined evidence must reach this score before the system treats the result as meaningful.
Action
Follow-up
ConditionalAdditional settings appear only because the selected action creates downstream work.
File a bug
Detailed controls appear when they become relevant instead of competing for attention on the main page.
Static interface reconstruction. Values and labels are illustrative.
Early feedback could have been interpreted as a collection of UI problems.
But better labels and spacing would not have solved the core issue. The hierarchy itself was flat.
I reorganized the experience around the causal sequence of the system:
That created clearer boundaries between selecting a detection model, understanding its current state, configuring scoring, defining a threshold, and configuring downstream automation.
The controls did not become less sophisticated. Their relationships became more legible.
Architecture comparison
Same capability. Clearer decision model.
Early
One long configuration surface
Repeated field patterns
Parameters exposed simultaneously
Automation mixed into detection logic
Refined
Choose how evidence is gathered before exposing deeper configuration.
Summarize weighted evidence instead of repeating every field.
Separate the decision rule from the scoring controls.
Reveal automation only after the selected action requires it.
Detection model selected first
Advanced logic can run in parallel
Current state summarized at the parent level
Detailed editing moved into focused flows
The redesign shifted from exposing system parameters to revealing the decisions those parameters control.
Reflection
Complex developer tools often need complex capabilities. That does not mean the interface should reproduce the underlying system architecture literally.
The strongest design move was identifying the decisions buried inside the configuration model and giving each one a clear place in the experience.
