Tag Archives: algorithms

I, algorithm: Can data-driven decision-making lead to dumb results?

Data-driven decision-making is more than just business intelligence dashboards or personalized recommendations in your Netflix queue. It’s now impacting your chance at scoring a new job, finding a date, or a signing a lease on an apartment. While big data may be helpful when you’re trying to find the best deal on a plane ticket, is it ever unfair in a way that hurts people? As IT leaders, when do we need to stop looking at the data and focus on the person?

For one anonymous Canadian applicant, a rescinded job offer based on his credit score was viewed as an absurd penalty for being “out of work for a while.” In an interview with The Huffington Post, the Ontario resident expressed frustration that his potential employer thought his credit score was an indicator he’d steal money. Get this: TransUnion government relations director Eric Rosenberg has admitted there’s no existing statistical correlation between poor credit and the likelihood to commit fraud.

Big data is an amazingly valuable tool for finding the right book on Amazon or getting the right dosage of antibiotics from your doctor, but are we beginning to see its limitations as a form of intelligence? Data-driven decision-making in the real world has both positive and negative aspects, especially where human romantic and socioeconomic relationships are concerned.

Cracking the code on romantic compatibility?

For the 25 percent of Canadians who have dipped their toes into the online dating pool, big data is playing a bigger role than ever before. Chances are, your matches are ranked by perceived compatibility, based on how you’ve answered a series of questions about your hopes, dreams, and feelings about horror films.

While it may seem strange to apply algorithms to the highly chemical and inexact science of attraction, some proponents insist it really works—to an extent, anyway. When CEO Amy Webb successfully “hacked online dating” with a 72-point set of criteria to discover compatible matches, she discovered that there’s more to romance and attraction than she ever thought.

Algorithms aren’t biased—but they don’t look for hustle

For IT pros, hiring algorithms are an application of big data that may be encroaching too close to your personal world for comfort. The explosion of “people analytics” means that data-driven hiring decisions are no longer reserved for elite organizations like Facebook. Supporters of this approach argue that it can lead to objectively better and more diverse decisions than if a human hand-picked an applicant.

While employers may be tempted to use mathematical models to remove human bias from the equation, thinking of algorithms as impartial is a mistake, according to mathematician Cathy O’Neil. While hiring algorithms may be capable of removing the dangerous human biases against the best candidate, not everyone is sold on the idea. Founder and chief executive of Millennium Search Amish Shah told The New York Times, “I look for passion and hustle, and there’s no data algorithm that could ever get to the bottom of that. It’s an intuition, gut feel, chemistry.”

When big data at work gets a little sketchy

Sure, data is revolutionizing the professional realm in tons of positive ways, like energy-efficient smart offices—but there’s still plenty of controversy. Biometric data collection on employees is one area with hotly contested ethics. From HR’s perspective, knowing the percentage of employees at-risk could be incredibly helpful. But where does the line between data-driven insight and personal privacy fall?

Some experts believe that biometric technology may be moving faster than companies know what to do with. The world of fitness trackers and genetic samples has been a total whirlwind. The Privacy Commissioner of Canada warned against this trend in 2016. The Commission’s report is clear: “An organization needs enough information about an individual to authorize a legitimate transaction, but needs to ensure that it does not collect, use, retain, or disclose personal information that is not necessary for that purpose.”

Data-driven decision making: Proceed with caution

Data has limits. This doesn’t mean you should unplug your Hadoop cluster, or ask your doctor to hand-crunch the numbers for your laser eye surgery. But IT pros need to understand the limitations of big data decisions in the real world, and use this filter for smarter and more ethical choices at work.

Google Flu famously failed to predict influenza outbreaks. This doesn’t mean that algorithms can’t ever work for epidemiology—it just means it’s still a work in progress. Any new or old artificial intelligence has flaws because it’s driven by humans, who are prone to mistakes. But algorithms also fail to account for the law of entropy or any factors outside their programmed logic.

Data-driven decision-making algorithms perform better in a wide range of concepts. They can be an effective tool for removing bias and error, but IT managers need to recognize that technology is only as smart as its human creator, and misapplications of artificial intelligence can lead to some really dumb decisions.

The secret formula behind Spotify and Tinder’s scary-accurate recs

You may not believe in soul mates, but Spotify’s Discover Weekly playlist is pretty close to true love.

When digging into the UX and algorithms of today’s best apps, Spotify is a prime example of brands that really know their customers on an individual level. The Discover Weekly playlists, which refresh on a weekly basis, allow users to discover new music without skipping through irrelevant tracks. Using algorithms to generate recommendations isn’t a new concept, but this is different from your average genre-and-tone-based playlist built by a machine.

The depth of accuracy induced comedian Dave Horwitz to tweet: “It’s scary how well @Spotify Discover Weekly playlists know me. Like a former-lover-who-lived-through-a-near-death experience-with-me well.” That level of custom experience is what we’re all striving for in design thinking. Seriously, what’s the Spotify secret, and how can you steal it?

Tinder comes to the stage with a similarly brilliant type of algorithm coupled with gamification, all while taking away some of the potentially unpleasant feelings about online dating thanks to all those mutual Facebook friends it shows you and your match already share. Tinder’s engineers have attraction down to the numbers, something that’s definitely the envy of all developers.

Innovation where UX and algorithms meet

There are fundamental flaws in algorithms that aim to serve as a replacement for human selection and judgment. Algorithms are only as smart as their rules and input, and they don’t know how to filter out the garbage unless they’re told how. That’s why data science-powered diagnostic tools haven’t replaced doctors. Algorithms don’t operate with the nuance of people yet, but companies like Spotify and Tinder are trying as hard as they can to make this a reality.

Innovation in design occurs when organizations get to know their users better than anyone else and use this knowledge to build products. It’s the bleeding edge of design thinking, which is both informed and improved by data. The best customer-facing apps have excluded your kids’ favourite artists from your custom playlists, while connecting you to your new favourite singer-songwriter.

1. Spotify

Spotify goes a lot deeper with classification than many other apps. They’re also not shy about pushing the bar further than their competitors. Spotify’s Matthew Ogle told Quartz, “If you’re the smallest, strangest musician in the world, doing something that only 20 people in the world will dig, we can now find those 20 people and connect the dots between the artist and the listeners.”

Finding odd patterns in the relationships between product and customers isn’t new. Way back in 2004, Walmart discovered that strawberry Pop-Tart sales spiked when a hurricane was about to hit. And Spotify uses many of the same tools Amazon and Netflix use to generate recommendations, like clustering techniques and probability.

The kicker is that Spotify uses natural language processing on music blogs to understand cultural context around artists and live events. This is what differentiates their technique. They’re applying deep learning techniques to analyze the way their tracks sound. These insights form a rapidly growing number of specific subgenres, like new Americana, alt country, or stomp and holler. These algorithms operate in real-time, creating playlists that speak to your soul. They’re also sensitive enough to pick up on the fact you’re really into “chamber pop” this week.

While Spotify’s engineers were a bit coy in a recent presentation about what’s next for their users, they plan on adding a feedback loop to their selection process. By integrating data on the songs that users skip and save from their Discover Weekly playlists, Spotify can only make your recommendations—and their super-sharp algorithm—better.

2. Tinder

As one of the world’s most popular dating apps, Tinder’s ease of use makes high-volume online dating fun. There are some hints of gamification; users’ swipes are interrupted by a micro-interaction when they receive a new match. Its instant feedback lets you know you’re likeable or, at the very least, swipe-able. Online dating can feel sketchy, but Tinder even makes room in their super-clean UI to show you mutual Facebook friends with potential matches.

Tinder isn’t just a really smart, addicting design. It’s also a data-based approach to romance that wants to know our behaviours perhaps better than we know ourselves. Every Tinder user has an “Elo score,” a complex number revealing where you fit into their community. Elo is based on real-time feedback from their user base. Tinder wants you to feel good, so you’re fed users with a similar Elo score who may have a similar personal style, sense of adventure, and other characteristics.

One of Tinder’s most recently launched algorithmic features is “smart photos,” or complicated testing of your profile pictures through a constant rotation to determine which are most attractive. Using a method called the Epsilon Greedy, it determines the swipe-attracting value of each photo on a scale of 0.1 to 1.0. Tinder’s engineers are convinced smart photos will land you more dates.

Tinder may not have cracked the whole code to algorithm-assisted compatibility and match potential, but they’ve come close. Is anyone else amazed there’s a massive database containing almost all the answers to the science of human attraction out there?

Human preference is complicated

“Taste” is never simple, and one of the key characteristics of any algorithm that really gets its users is understanding how to filter out the bad data. It’s apps with enough nuance to understand that the yacht rock you played at your dinner party last week sounds nothing like what you listen to while jogging. While Tinder’s Elo score is a closely guarded secret, they’ve managed to assign numeric values to the zeitgeist of human attraction. Before you know it, tomorrow’s user experience will be powered by algorithms like these that can discern between the data that’s “really you” and the data that needs to be discarded.