Hands on the Wheel Ch. 02

Here's Chapter 2 (finally!). If I had been this bad at estimating schedules when I was a freelance tech writer, I'd be living under a highway bridge in Plaquemines Parish, Louisiana. But once again, my characters gave me the one-finger salute and did what they damn well pleased. So mea culpa, y'all, mea maxima culpa. I promise to try to do better.

Ivan powered up his cell phone as soon as he landed at SFO; it immediately lit up and sounded off with a Presidential Alert: Shitstorm here. Call NOW. It was from Brian, of course; he bragged about hacking the Wireless Emergency Alert system but had never used it on Ivan. Ivan hit Brian's speed dial.

"Ivan! Where are you?"

"Wheels down at SFO. What's the shitstorm?"

"Kimberly's got a nasty bug, randomly fucking up responses. Half a dozen of the beta sites so far. We've put together a search-and-destroy team, you, me, and Jean FitzHenry."

"Jesus Brian, I'm just getting off an 11-hour flight—"

"You're booked on an American red-eye to Boston in..." Ivan heard Brian typing on his laptop. "...ninety minutes. Check in online and download your boarding pass to your phone. You should be able to get through Immigration and Customs in plenty of time at this hour. Don't worry about your checked luggage, just bring your carry-on and laptop; we'll have somebody pick up your luggage tomorrow. You can buy more skivvies at Wal-Mart." He said all that without pausing to breathe; it was obvious there was no point in arguing.

Ivan couldn't see any sleep in his near future. "Wal-Mart my ass! If I have to buy underwear I'm going first class. Tarzhay, baby!" When Brian didn't laugh, he knew the problem was really serious.

"Look, Ivan, there's a lot riding on us. We've got to find and fix this fucking problem as soon as we can. The VCs are getting antsy and Jeremy's sweating bullets. The whole company might be riding on this. Try to sleep on the way, you're going to need it."

Ivan knew when he was beat. "Okay, okay. Got your toolkit?"

"Of course, bro, never leave home without it."

"Good. I've got my laptop, I'll download the test suite and anything else I need. See you at Logan."

Brian's toolkit was a collection of apps, some he wrote himself and some he gathered—not all from totally legitimate sources—that let him look at and even diddle with networks and computers whether or not he was an authorized user. The tools could be used for nefarious purposes, but as far as Ivan knew he used them strictly to support Golkonda's development and testing needs.

The test suite was what Brian, Ivan, and the director of marketing believed was a representative sample of the sort of projects customers would use Kimberly for; they ranged from unrealistically simple to really convoluted. The databases were encrypted in the cloud. He also downloaded the anticipated results—it was well-nigh impossible to define a "correct" result in data mining, especially when you throw AI into the mix. The test suite databases were sufficiently skewed to produce the anticipated result every run.

When Ivan got to the gate, he opened his laptop. He listened to voice mails from his brother in Colorado and sister in Illinois; he'd call them from Boston. He didn't even open the rest of the text messages. They could wait until the 5-hour flight to Boston if the plane had wi-fi.

He started getting antsy when boarding started and Brian and Jean weren't at the gate yet; they got there, out of breath, with three minutes to spare. Brian was angry; no, he was livid. "Fucking TSA," he wheezed. "I swear to God they jones on making you miss your flight. And if you complain, you might wind up on their fucking no-fly list!" He handed me my boarding pass. "You're in first class since you've been flying for days; Jean and I are back in steerage."

They dragged their carry-ons onto the jetway, the last to board. As Ivan put his bag in the overhead bin, he could hear Brian bitching to a flight attendant about having to stow his bag several rows away from his seat as if the inconvenience was a violation of his civil rights. Ivan was relieved when he shut up before they called the airport police to toss him off.

Ivan was concerned as he strapped into his seat. Brian wasn't usually that obnoxious; his stress level had to be off the charts. The problem obviously was serious, but he was too tired to worry about it. He fell asleep before they reached cruising altitude and slept fitfully most of the way to Boston. A bathroom call woke him up once, then a troublesome dream a bit later; he drifted back to sleep both times, fretting about how to find Fumiko. He resolved to start trying as soon as they got to their hotel.

_________

Data mining is the catch phrase for the search for valuable nuggets of information buried in heretofore undiscovered patterns or relationships in terabytes of often seemingly unrelated data—in a beery rant following a particularly unsatisfactory meeting with marketing, Ivan called data mining "a misleading metaphoric moniker slapped on by hucksters". Marketing mavens were delighted to polish its image as the latest holy grail for companies searching for ways to prosper, whether they needed another flashy play for Increased Shareholder Value or a means of survival, to avoid being brushed aside by competitors with better products or VC-fueled startups driven by young wizards working hundred-hour weeks.

First-generation data miners needed crazy-expensive supercomputers. Second-generation systems brought the cost down a lot, but were still so expensive that most boards of directors were reluctant to invest so much in something not guaranteed to produce an acceptable return. Third-generation systems were supposed to bring the cost down almost to impulse-purchase level—impulse purchases for businesses and other organizations that didn't suffer from a penchant for penny pinching, that is—but the anticipated breakthrough always seemed to be at least a year off.

Golkonda's first product was Argentum, a data mining system with a local front end and back end in the cloud. Thanks to some proprietary pre-processing in the cloud, the final steps of analysis and formulation of responses could run on a cluster of two or three servers. It made data mining affordable for mid-size companies and institutions, but few were ready to take the risk of going with an untried company and product.

A year later came Auricle, which was faster, could handle larger databases, and still ran on the same (comparatively) minimal system as Argentum. Another company in the market—well, more accurately the market leader, led by its ambitious, aggressive founder—took great exception to the name Auricle, filing suit claiming it would "confuse all potential customers in both the database and cloud computing markets." To avoid expensive litigation, Golkonda relaunched the product as Aurum.

Those at Golkonda familiar with the politicians' mantra "I don't care what you say about me, just be sure to spell my name right." welcomed the ensuing publicity. Some cynics even suspected that the allegedly confusing name was chosen deliberately. Thanks to the public exposure, Aurum was somewhat more successful than Argentum, although it never earned back its development and marketing costs (including, of course, the expense of changing the name).

But both Argentum and Aurum were actually stalking horses for the third product, Kimberly; they admirably served their purpose of giving Golkonda visibility in the data mining market. Kimberly was what Jeremy intended all along to be their flagship product. It was remarkably faster than all existing products, ran on much smaller systems, and could deal with multi-petabyte data sets. It built a first-pass index in the cloud, and performed one of Jeremy's innovations called relation clustering and pruning. These results were then encrypted and sent to the local servers, where final processing took place. The index was deleted from the cloud when the local cluster reported receiving all the data.

The final processing included another innovation of Jeremy's, an artificial intelligence module that made inferences based on the nature of the dataset, the specified goals submitted as part of the project, and any history it had of previous projects submitted by the owner. It then built (or refined) its own model of what sorts of intelligence the owner typically looked for. Marketing literature called this heuristic goal seeking. As the AI model grew more detailed (and, it was hoped, more accurate), Kimberly could actually propose further projects aimed at similar or related goals.

Golkonda negotiated confidential non-disclosure agreements with six companies carefully selected to represent the most important potential market segments, then sent engineers to install beta versions of Kimberly. For more than a week the response was enthusiastic, generating much valuable feedback, breathless buzz in the blogosphere, and optimism at Golkonda. It promised to disrupt the market the no less than Snapchat or Spotify. Speculation about an IPO began.

Then the shitstorm hit: Kimberly started going bonkers.

_________

Their flight landed in Boston just after 6 am. They had no checked luggage, so they went directly to the car rental. By 9:00 they had checked in to the Marriott Newton at Mass Pike and Route 128, taken quick showers and changed clothes, gobbled a hasty breakfast (with copious cups of coffee), and headed for Patriot Partners, a hedge fund.

After meeting with their IT staff, the news was even worse than they feared. Kimberly never went tits up, but sometimes it returned incorrect results. Sometimes. A few runs might work perfectly, then one or more would return spurious results; rinse and repeat. The scary part was that the results weren't obviously wrong, but off just enough that the results could be disastrous if they were used to guide a decision involving a venture that was highly leveraged—which therefore possessed a truly ugly downside to match the seductive upside—which just happened to be the stock in trade of Patriot Partners.

The IT staff's opinion of Kimberly was so poor they had quit running it—not even using the canned projects in the test suite—and had quarantined the system where it was installed, taking it offline from both the internet and their own intranet. A later meeting with the managing partners was even more chilling: Patriot Partners considered Kimberly not just a poor tool but a toxic one. Not exactly what you want beta sites whispering to tech industry reporters and bloggers.

Bolstered by three separate phone calls from Jeremy and a masterful demonstration of PR skills by Jean FitzHenry that included a stirring appeal to their reputation for entrepreneurial chutzpah—augmented by her occasional soulful glances and impressive deep breathing exercises—they managed to convince the partners to hold off on formally ending the beta test and going public with their reasons for twelve days (they had asked for two weeks).

While Brian tried to reassure Patriot's IT staff that they would fix the problem, and Jean tried to soothe the concerns of the partners, Ivan was granted limited access to their test system. He chose a project from the test suite about midrange in complexity, cleared all other non-supervisor tasks to ensure identical operating environments, then ran the project four times in a row.

Each run should have yielded the same result, but each result was different and none was "correct." He ran it four more times; this time the first was correct, the second was not, and the last two were. Random errors occurring at random intervals. They had 12 days to fix something they might not even be able to find, let alone fix. Ivan broke the news as they drove back to the Marriott.

They gathered in the hotel bar for a drink before dinner, then Brian called Jeremy and scheduled a Skype conference at 8:30. They ate a hurried dinner (sans wine, to keep a clear head) and gathered in Brian's room. He set up the Skype link on his laptop; Ivan again acted as bearer of bad tidings, then they took turns describing how they handled the situation.

Jeremy usually relied on others to come up with solutions, but this time he took charge. He talked about the importance of Kimberly: "We're counting on it to be our biggest cash cow. Corporate PR started a quiet blitz (Ed: oxymoron alert!) a week ago. There's some buzz starting in the trade mags about a possible IPO, but if we don't find and fix this bug in Kimberly the IPO is SOL."

Ivan was surprised, almost shocked, to hear Jeremy spouting business jargon. Then he realized that while he and Brian had been locked in the design/coding zone for the last couple of years, Jeremy had been fighting to keep the company afloat by consorting with vulture capitalists who wouldn't hesitate to pull the plug if they thought Golkonda didn't have a good chance to be another Silicon Valley unicorn.

He told Brian and Jean they should visit each beta site as soon as possible, Brian to placate their IT folks and Jean to soothe the poobahs. Brian had the tech credentials to get the IT mavens to listen to him, but Lakshmi Jeanne Maureen FitzHenry—she preferred to spell her name Jean because she got tired of people calling her Jane or Jeannie or JeeAnne—didn't know SQL from squim. She did, however, know how to persuade; legend had it that she could convince skeptical VPs that Kimberly would probably find surefire sales prospects (or lucrative research grants) in nothing more than a few billion of last year's horoscopes and losing lottery tickets.

Their first stop was Worldwide Communications Partners in McLean, Virginia. Ivan was to stay in Newton, find the bug, and kill it. Jean found a red-eye to Dulles; she and Brian checked out at 9:45 and just made the flight.

_________

Jeremy Levinson and Brian Green had been high school classmates and casual friends in Palo Alto. Upon graduating from Paly, Jeremy opted for Cal Tech to get a BS in physics and MS in math, then back to Palo Alto for a PhD in information theory at Stanford. Brian spent the summer smoking dope and playing World of Warcraft, then said to hell with it and joined the Air Force.

Jeremy's dissertation, Modeling Database Search Performance to Optimize Query Structures and Strategies, was impressively larded with abstruse formulae, dense tables of supporting statistics, and seemingly significant conclusions. It was enthusiastically accepted, but made no reference to his startling—if somewhat accidental—discovery that database manipulations could be speeded up (or host system requirements reduced) at least two orders of magnitude if each data point were assigned a unique random number.

Unfortunately, some databases ripe for data mining had trillions, if not more, data points. Most attempts to generate a large series of truly random numbers required monitoring electron emissions from a heated object or paying a small army of minimum wage workers to record the result each time they rolled a die or plucked a numbered ball from a rotating cage. Disappointingly, many such sequences proved not to be random under scrutiny because of unintentional or unanticipated biases.

But Jeremy solved the problem without resorting to dicey physical processes. The classic source of random numbers was the 1955 Rand Corporation book A Million Random Digits with 100,000 Normal Deviates. While working on his master's thesis at Cal Tech, he had played around with trying to permute those million into a longer series and came up with an approach he thought might work; he had stopped working on it when the deadline for thesis submittal loomed.

He dug through his notebooks, found the pertinent entries, and picked up where he left off. It took him a few weeks of fairly intensive trials, but finally he could extrapolate additional verifiably random digits (not pseudorandom) from the million originals. He suspected that the number of random digits he could extrapolate might be limited to a million to the millionth, but never bothered trying to prove it.

Using the modeling exercises from his dissertation, he designed and coded a prototype of a data-mining system that was small enough to run on a cluster of two or three servers. The database could be stored either locally or in the cloud; the critical random number ID was assigned as each element was accepted from the database. When the prototype was more or less stable, Jeremy got his doctoral advisor to wangle an appointment with a venture capitalist. He pitched, demonstrated, and won the first round of seed funding. Golkonda was born.

In the meantime, Brian had completed his Air Force service. He tested well enough in basic training to be sent to computer security school at Keesler AFB in Biloxi, then to Goodfellow AFB in San Angelo for more training in intelligence and cryptologic analysis. Along with his technical skills, he picked up a taste for Cajun and Tex-Mex food and country music, and some unconventional computer skills usually described as hacking (although he insisted that he always wore a white hat—well, perhaps occasionally a shade of gray).

The Air Force was sufficiently impressed with his skill and work ethic that they offered to send him to Officer Training School at Maxwell AFB in Montgomery if he would re-up, but Brian didn't want to spend any more time in the South, he wanted to go home to the Bay Area and dress however he chose. Besides, Jeremy had just got his doctorate at Stanford, was getting ready to start a company, and wanted Brian to join him as Chief Technical Officer and Director of Security.

The pay would be twice what he had been making in the Air Force, and he also got a piece of the company. If he felt culturally deprived, he could go to night school at San José State (Snatdorf was just too bloody expensive), but instead he felt romantically deprived.

He met Vicki Douglas at a nearby Starbucks not long after Golkonda opened its doors; she was a grad student working toward an MS in statistical psychology. He was so smitten, he stopped cruising singles bars, uninstalled Tinder and Snapchat, and by the third date had plighted his troth. They were married within a year. In the next few years, Jeremy would be delighted to discover that married life somehow inspired heretofore unsuspected management skills in Brian that almost matched his technical chops.

About the same time Brian came onboard, Jeremy went back to his hometown to attend a friend's wedding. He came back with a third employee, a woman named simply Woodley, who was Jeremy's Administrative Assistant and Interim Director of Human Resources. She was solemn, never smiled, always dressed in black jeans, long-sleeved bulky black turtleneck (jersey in summer, sweater in winter), and black Doc Martens.

Not long after that, Jeremy, Brian, and Woodley hired a software engineer from Iowa State named Ivan Wolfe. Golkonda had not only opened its doors, it was off and running.

_________

The morning after Brian and Jean left the Marriott, Ivan started running into roadblocks at Patriot partners. They had shut the test system completely down, and it took almost two days to get them to bring it back up and online, then authorize him to operate it without supervision. He spent the next day—actually 19 hours after he convinced them to let him stay after the IT staff went home—convincing himself that Kimberly really was behaving as squirrely as Pilot claimed.

That glacial progress prompted Ivan to call Woodley and ask her to have his production laptop overnighted to him at the Marriott. It was a tweaked HP Omen gaming system, with a majorly overclocked 8th Gen Intel Core I7 processor/Radeon graphics processor, 64 gigs of RAM, 4TB SSD, and all needed I/O ports; it could run the full production version of Kimberly. He ran it continuously on AC power, because the battery wouldn't last much over two hours, and taking the battery out reduced the weight and reduced the risk of the lithium battery getting cooked by all the radiated heat. He asked her to make sure the most recent distro was installed, and told her that he was going to do all testing on his laptop from now on to avoid the bureaucratic delays at Patriot partners.