Last major update on 24-9-2024 22:35 hrs
Disproportion: A Hidden Gem in Problem-Solving by Anomaly Detection
Anomaly detection by disproportion saved millions in my project. Learn how this technique can uncover hidden inefficiencies and solve a complex problem.
Imagine a data center facing a seemingly insurmountable challenge: expanding its already huge power plant at a late stage of installation, when finding additional space was well nigh impossible. This was the problem that confronted a team of engineers, and their solution was revolutionary. By using anomaly detection through disproportion, a hidden gem in problem-solving, they were successful in proving the additional power plant requirement erroneous and saved millions. At first, it seemed unbelievable, but this breakthrough proved that even the most complex problems can be solved with innovative thinking.
The Power of the Principle of Disproportion: A Revolutionary Anomaly Detection Technique
The principle of disproportion offers a unique approach to anomaly detection, transforming how we approach problem-solving. Whether in nature or engineered systems, we expect things to follow a balance in proportion, sometimes even being unaware of it. When one such critical proportional balance breaks, it signals a major potential inefficiency. It raises an alarm urging us at least to deeply examine it for detecting presence of potential problem waiting to surface.
Disproportion serves as a lens through which these anomalies can be detected and addressed, providing a new way to uncover inefficiencies that might have otherwise remained hidden. By spotting these subtle imbalances, we open the door to smarter, more innovative solutions to complex challenges.
Everyday Proportions: Examples We Encounter Daily
1. Human Body Proportions:
- Head to body ratio: The size of a person’s head is generally proportional to their height.
- Arm to leg ratio: The length of arms and legs is usually proportional to overall height. A fable goes: the person who can encircle the Iron Pillar of Qutab Minar at Delhi will get whatever he wishes for. It is rare for a person to achieve it because arms need to be extra-long.
2. Architectural Proportions:
- Golden Ratio: This mysterious mathematical ratio, often found in nature and art, is also used in architecture to create visually pleasing structures. It’s embedded in almost every architectural or digital design. Any deviation from this ratio in a design will jar your senses.
- Symmetry: Buildings are often designed with matching, symmetrical elements for aesthetic and functional balance. A room where the walls do not meet the floor at perfect 90-degree angles has the potential to create severe mental ailments in the persons living in such an unholy room. I lived in one, and I know its effect on my wife.
3. Health and Fitness:
- Body Mass Index (BMI): This ratio is used to assess a person's weight in relation to their height and is a common indicator of health. Deviation from the recommended BMI index should point to a need for correction of lifestyle or food habits for better health and longer life.
4. Engine to Car Body:
- A car's engine should typically occupy around 10-15% of the car's total interior volume.
5. Battery and UPS Size in Relation to Server and Disk Space:
- In 2008, the size of laptop batteries typically occupied about one-sixth of the overall body space, with a direct relationship to the motherboard and disk size, where the two together occupied around two-thirds combined.
- In larger setups, such as data centers, the battery and UPS systems typically required floor space of about 25-30% of the combined size of the floor space dedicated to servers and disk systems.
- Essentially, the battery systems can no way be larger than the motherboard and internal disk in a laptop or the server and disks in a large data center.
A Case of Real Life Problem Solving: Expansion Order for the Already Large Power Plant of a Sophisticated Data Center
We were closely involved in building a large data center at the time. Site preparation progressed stage by stage, floor by floor. It was quite an advanced stage. Rows of computing and storage resources were all in their appointed place.
As was the practice, being the heaviest, the battery and power plants were housed on the ground floor. The unending stacks of power equipment surprised us all. We never imagined that so much power would be required to run the main equipment. With quite a bit of difficulty, we managed to make space for the power equipment.
At this point, we received an order from HQ for immediate expansion of the already bloated power equipment, and we were shocked. Being disciplined soldiers (the installation felt like a war, after all), we started wracking our brains about how to squeeze out the extra space needed. Space was at a premium on the ground floor, and, on top of that, a continuous hall was needed.
This was a multi-vendor complex installation, and someone somewhere had made a recalculation of the power requirement, leading to this expansion proposal at such a late stage.
Less Than Optimal Solution for Power Plant Expansion
After multiple passes at the problem, we managed to arrive at a less than optimal solution that left me deeply dissatisfied. It involved breaking down a few walls for extra space, but the solution felt awkward and ungainly.
This created a niggling discomfort in my mind, and over a week, this discomfort would surface as a rough-edge (explore how rough-edge detection technique works).
Visualization of Superimposition of Floors in Stacked Layers: Uncovering the Incongruity
One fateful day, while walking along the ground floor corridor, I suddenly visualized all the floors superimposed in my mind, one on top of the other, like stacked layers. This visualization struck me like a streak of lightning—a sudden and seemingly accidental insight born out of deep discontent and my unusual problem-solving abilities.
Within minutes, the glaring disproportion became clear: after the expansion, the power plant space would be the same size—or even larger—than the space for the main computing equipment it was meant to power!
The principle of disproportion, although I wasn’t aware of its formalism at the time, confronted me with this huge incongruity.
Validation by Chief Engineer: Proof of Requirement Error and Proportional Violation
I am no expert in electrical matters. I called upon our Chief Electrical Engineer, explained the situation briefly, and requested him to review the power requirement in depth. His curiosity was also piqued. Being both thorough and an expert, he produced a three-page paper in two days, irrefutably exposing the serious error in the new requirement assessment that led to the violation of proportional balance.
Convincing HQ: A Critical Turning Point
We needed to convince HQ. I sent a brief letter to HQ, attaching the three page paper, explaining the error in power expansion order. The pro-active Director-in-charge in our Board immediately formed an organizational expert committee that included all Chief Electrical Engineers in the country. Their split-hair dissection of the problem, from an unimaginable number of angles, sealed the fate of the awkward power expansion proposal within a week.
It was a swift corrective action that saved us millions in both capital and recurring costs.
The one moment of revelation of the disproportionality saved not only our data center but all four regional centers spread across the country from becoming inefficient power-guzzlers, a tragedy that would have impacted operations for years to come.
The Principle of Disproportion in Action
All of this was due to the chance application of the principle of disproportion—a vague and abstract idea buried in my chaotic pool of conscious knowledge. While requesting our electrical chief for the in-depth review of the power requirement, I simply asked, “Can you imagine a laptop battery being larger than the processor chip?”
End Note: The Principle of Disproportion as Anomaly Detection
Upon reflection, I now understand this principle as a special case of the more general “Anomaly Detection Technique” or "Spot the Odd Technique", routinely used in R&D, criminal investigation, software debugging, and any type of fault-finding system such as mobile network troubleshooting or healthcare. Rough-edge rejection technique is similar to this class of problem solving techniques tied together by the common thread of anomaly detection. But, the methods associated with the techniques and their application areas may vary widely. On the other hand, visualization problem solving skill is an extremely important skill that plays a significant role in many instances of complex problem solving.
Even without knowing the name of the principle, the alert and inquisitive mind will always seek out the unusual, which may hold the key to breaking down the problem or, at the very least, hinting at a lurking issue.
The principle of disproportion is one of the vaguest types of oddities, yet it holds the promise of yielding great results if exploited to the fullest.
A Representative Data Center Floor Layout
In a modern day data center, because of rapid advancement in battery and power plant technology, the footprint of the batteries is much less compared to the servers. This was true as well during the year 2008-2009 when the crucial disproportionality was detected to solve an unnecessary power bank expansion.
Valuable Reading Resources on Proportional Balances
- 9 Examples of the Golden Ratio in Nature, from Pinecones to the Human Body : Explores what the golden ratio is and how it shapes the intricate patterns of plants, animals, and even our own bodies—an eye-opener to the power and reach of proportional balances in nature.
- Scale and Proportion in Architectural Design : In the world of architecture, understanding the concepts of scale and proportion is crucial for creating visually pleasing and harmonious designs : A guide to proportional balances in Architectural designs.
- Ideal Body Proportions : Ideal human body proportions are often used in the fields of health and fitness, fashion, and even art, to determine what is considered aesthetically pleasing or physically optimal. A valuable resource for personal use.
- Understanding Disproportionality Part 1: What Disproportionality Is and Why It Matters | 2019 : The starting paper in a four part series on disproportionality in US Public School Education System.
- What We Mean By ‘Disproportionate Impacts’ : A critical discussion on how Climate Crisis disproportionately affects low-income communities and communities of color.
- Leg Length, Body Proportion, and Health : A very interesting treatise on health implications of disproportionate human body parts.
- Detection and explanation of anomalies in healthcare data : This is on anomaly detection in healthcare data. It is disappointing to find such an important broad-range problem solving concept as anomaly detection getting all attention in the area of datasets, but not in other areas of activity.