Industry News & Trends

According to Gartner, “by 2024, 60% of the data used for the development of AI and analytics projects will be synthetically generated.” I don’t know about you but I was shocked by that statistic and it shows that synthetic data is definitely a rising trend.

In this blog post I will be exploring synthetic data to find out:

• What is synthetic data?
• What’s so great about it?
• What’s no-so-great about it?
• What techniques are used to generate synthetic data?

What is synthetic data?

Any data generated by a computer simulation and not collected from real-world signals is known as synthetic data. Usually, an algorithm is trained on real data and is able to reproduce the same statistical properties of that dataset in a new, synthetic sample.

A classic example of synthetic data can be found in flight simulator games that attempt to mimic real-life events within a controlled environment.

Importantly, synthetic data can make new advances in AI possible and can enhance the decision-making process in businesses.

What are the benefits of synthetic data?

The development of cutting edge, successful models in AI and ML nowadays requires larger and larger volumes of high-quality data. Synthetic data is used across many different industries and can be particularly useful in these cases:

• Preserving privacy by generating a synthetic dataset without any sensitive information. This is particularly useful in the healthcare and financial sectors.
• Build complex models that require large amounts of data that are either too expensive or too time-consuming to collect (such as with self-driving cars, or other computer vision applications)
• Researchers are able to explore and test new algorithms under controlled conditions with synthetic data.

What are the challenges with synthetic data?

Naturally, there are a number of challenges that come with generating and using synthetic data. I’m sure as more research is conducted in the area of synthetic data, these challenges will be easier to overcome in the future.

• Synthetic data is only as good as the underlying model used to generate it. This would be a classic example of garbage in, garbage out. Any biases found in the original data will carry through into the synthetic data.
• Generating synthetic data is a very complex, time-consuming process and requires highly skilled individuals to build the algorithms that generate the data.
• Depending on the application of synthetic data, there could be some serious ramifications if the models that are built on synthetic data go wrong.
• Business users or researchers may not be very trusting of synthetic data since it is still a very new area.
• If data is generated using machine learning models then overfitting can lead to synthetic data that does not generalise well to real-world scenarios.

Techniques for generating synthetic data

• Random sample generators: Scikit-learn has a library for generating datasets at various sizes and complexities.
• Fitting to a known distribution: Monte Carlo method
• Decision tree machine learning models
• Generative adversarial network (GAN) – see this case study about how American Express used GAN’s to generate synthetic data that help make their fraud detection models more accurate.
• Domain randomisation – altering images in a way that improves neural network models (such as changing the size, lighting, or colours in an image).

Synthetic data generation is being offered by more and more companies. Most notably is MIT’s Synthetic Data Vault which is an open source software ecosystem for generating synthetic data.

Further reading & references

• A comprehensive survey of synthetic data and its applications can be found in this paper.
• The ultimate guide to synthetic data by AI Multiple
• An overview of synthetic data by NVIDIA

Gartner released their top strategic technology trends for 2022 and Decision Intelligence was one of them. They described it as “a practical approach to improve organisational decision making”. It was the first time I had ever come across the term but it definitely got my attention. I wanted to find out more about it and how it could be used in practice.

To be honest, I found that to be a lot easier said than done. Decision Intelligence is still a relatively new field and there are some amazing pioneers out there such as Lorien Pratt, who wrote a book on the topic, and Cassie Kozyrkov, head of decision intelligence at Google.

However, I struggled to find actionable resources on the topic.

When learning something new, it is helpful to first understand its philosophical and theoretical components. However, you may get to a point, as I did, where you begin to ask yourself “great, this thing is important, how do I use it?” It is here that I found resources to be incomplete.

In this blog post, I will give a (brief) overview of decision intelligence and why it is important. Then I will do my best to explain how decision intelligence can be applied.

What is Decision Intelligence?

The basic idea behind decision intelligence is that “decisions are based on our understanding of how actions lead to outcomes“. It answers the question: “if I make this decision and take this action today, what will be the outcome tomorrow?”

Beyond that, making a decision is about forecasting through time in the hopes that the desired outcome can be achieved. Decision intelligence allows you to understand the ripple effect of a decision across other areas, both internally and externally.

Decision intelligence is not an alternative to current solutions or another tool in the data toolbox. Instead, it is a unifier: bringing together complex systems, business intelligence, analytics, machine learning, and AI into a framework where the decision is the main driver.

Instead of trying to find the tools and methods that store, retrieve, analyse, and support data, decision intelligence focuses on the decision first and the data takes on a supporting role.

Why is Decision Intelligence Important?

It is common to find a large gap between the decision-makers in an organisation and the data scientists and analysts. Decision-makers feel that they don’t have enough of the right information available to them to make big decisions that could have a significant impact on the business. On the other hand, many data scientists and analysts feel like their models, reports, and analyses aren’t being used to their full potential in the business.

Decision intelligence helps to bridge this gap, requiring a new way of thinking for both decision-makers and analysts. By focusing on decisions first and allowing them to be the driver of everything else, we can do a much better job of working together. Ultimately, models, reports, and analyses are not of any value (no matter how complex or cutting edge they may be) unless they lead to some action.

In organisations with so much at stake like the Millennium Project and the United Nations, there is a high risk of making decisions that can have unintended consequences. In these cases, the systems are so complex and overwhelming that decision-makers tend to suffer from ‘analysis-paralysis’, and rather than making data-driven decisions, they resort to just using their intuition.

With the rise of big data, we’ve got a new problem. How can we make the connection between data and the problems we care about?

How can data be used to solve really difficult problems like conflict and poverty? For a company, how can data be used to make more money and gain a competitive advantage? For medical providers in Sub-Saharan Africa, how can data help them decide where they should get medicine and which doctors they should train?

How to Apply Decision Intelligence

Through my research, I have found 2 main ways to use decision intelligence in practice. The exact methodology of DI is different for every organisation but usually involves a cause-and-effect diagram and a visual decision model.

The process starts with a decision in mind and by applying a blueprint for how to approach that decision, we can then determine what action needs to be taken to reach an outcome.

Causal Decision Diagram

The core application of decision intelligence is to understand how we can get from cause to effect. Lorien Pratt refers to the diagrams that allow us to examine cause and effect as a Causal Decision Diagram (CDD).

She had found a pattern in decision making through her work and created a template that contains the following elements:

• Decision Levers (choices) – the things you can change; decisions you need to make
• Externals – things that cannot be affected directly but are related to and affect the outcome
• Intermediates – factors involved in the decision
• Outcomes – the thing you are ultimately trying to acheive

Interactive Decision Models

The final output of a decision model is a visual representation of the causal decision diagram. This is the part of the decision model that incorporates data to objectively determine the effect of a decision.

The process of reaching a final decision model requires a combination of business intelligence, predictive analytics, machine learning, and various other analytical techniques.

Lorien Pratt gives a nice example of this visual model by considering the impact of the decision to invest in a new training program in a company. Levers can be adjusted and, together with some predictive models, the total investment benefit can be determined.

Decision intelligence is the missing link between data and decision-making. For all the amazing data science and artificial intelligence advances being made today, none of it is valuable unless it can be used to take action.

I hope this blog post was able to give you a sense of what decision intelligence is and how it can be used in practice.

Some of the biggest, most innovative tech giants are still struggling with gender equality in their workforce. Only 33% of the employees at Google are female and only 23% of the workforce at Facebook is female.

The Stats

Overall, only 26% of the data and AI workforce is female, according to the World Economic Forum.

Can you believe that we are going into 2021 and this is still such a significant problem in tech today?

The World Economic Forum actually predicts that it’ll take over 99 years to close the gender gap and achieve gender equality.

Taking a peek into the demographics at college and university can shed a little light on why the workforce is so unequal. Back in the 1980’s, 37% of computer science majors were women. Today? Only 18%.

Did you know that in the United States, women who worked in computer and mathematical occupations made 82 cents to the dollar  men earned in 2019? What’s more, this has barely changed since 2011!

Putting numbers and pay aside, it is also crucial to consider the harassment and sexism that women face in the workplace, as well as the hostility women tend to experience towards pregnancy and can even lose up to 4% of their hourly earnings for each child they have.

Action

OK, so what can we do about it? I believe the core to the solution is to empower and pioneer.

Change isn’t going to happen ‘eventually’. This isn’t one of those things that ‘needs time’. The only way to effect real change is to take action.

What does that look like?

• Empower: individuals can empower and support women both within and outside the workplace – read, learn, get involved, encourage the younger generations around you and on social media to pursue a career in tech, and change the dialogue around what tech roles look like.
• Pioneer: companies need to make conscious, strategic  decisions towards creating a more gender equal culture.

One of the greatest pioneers in the movement toward gender equality in the workplace is Boston Consulting Group (BCG), who have been named the best company for women by women. Can you believe that 50% of their workforce is female? Amazing!

To be honest, these stats and challenges are completely new to me, even though I am a woman in tech. I was fortunate enough to be raised by a mother who encouraged me to pursue whatever career path interested me. She was the perfect blueprint to what a strong, successful woman should look like to me. Not only that,  she was also a single parent to two kids! No small feat in my opinion!

It is only recently that I have been exposed to the stories, challenges and struggles that women face not just in tech but in many other fields, too. I want to listen to their stories and amplify their voices, driving change in any way I can.

Ultimately, it is companies like BCG and programs like TechGirlz, Girls Who Code and Brave, Not Perfect that are shining beacons of a future that feels so much closer than the 99 year prediction we’re being told, we just need the courage and commitment to take action!

If you want to learn more then I recommend you check out this list of 15 must-reads for woman in tech