Alumni Analyzer (a business line of TVB Associates) is pleased to share this statement from a happy client:
Alumni Analyzer’s findings were immediately useful in CLS government relations’ activities, enabling us to tell objective, data-driven stories about our impact on the training of highly qualified personnel.
Bill Matiko, CEO, Canadian Light Source
Alumni Analyzer’s results show that our alumni have a strong tendency to pursue higher academic degrees, to stay in Canada and in Saskatchewan, and to return to Canada if they study abroad. A strong majority of these alumni are employed in Canada’s top four most high-tech and research-intensive economic sectors. CLS alumni also achieve more or higher leadership positions over time. For example, after 10 years, CLS alumni are over 400% more likely to be in executive positions than the control group
TVB Associates is pleased to officially launch the Alumni Analyzer brand with successful case studies from two major research facilities: SNOLAB and CMC Microsystems!
Do you want alumni data to help you tell compelling stories to funders, potential recruits or employers?
Alumni Analyzer (a business line of TVB Associates) is pleased to share this statement from a happy client:
“SNOLAB is the world’s leading deep underground clean lab hosting a dozen scientific experiments by large Canadian and international research teams. During more than 20 years of operation, numerous students and post-doctorates have passed through our facility either as users or employees. Inspiring and training these alumni for careers is one of our principal impacts on society.
“As a facility that relies on public funds, having objective data on SNOLAB’s impacts is critical. Yet tracking our alumni through their careers has been a challenge. Alumni Analyzer (a business line of TVB Associates) found our alumni, connected with them, and documented their educational achievements and professional career paths – both statistically and anecdotally.
“Alumni Analyzer produced a comprehensive report detailing SNOLAB’s success in inspiring and training staff and students, and demonstrating the valuable role SNOLAB alumni play in Canada’s tech and innovation ecosystem.
“Alumni Analyzer delivered exactly what SNOLAB needed: objective, aggregate data showing SNOLAB alumni tend to pursue higher educational achievements, have high rates of retention within Ontario and Canada, enjoy strong career progression, and are more likely than other Canadian science graduates to work in high-tech and research-intensive sectors of the Canadian economy.”
CMC Microsystems, Canada’s leading hardware technology facilitator, and manager of Canada’s National Design Network® (CNDN), has shared results from its recent study on CNDN alumni, and the training of Highly Qualified Personnel (HQP) in Canada.
“CNDN researchers graduate as ‘walking technology transfer,’ bringing a holistic skillset into the marketplace,” writes CMC Microsystems.
TVB Associates used LinkedIn data to deliver key portions of their study, identifying findings such as:
Accelerated career progression: Alumni are about twice as likely to be in executive or management positions within 10 years, and about 25% more likely after 20 years.
Greater employee retention and engagement: CNDN alumni remain with their employers longer than peers who did not use CNDN, and this effect is seen for up to 20 years.
Retention in Canada: Over 70% of CNDN alumni continue to work in Canada for over 20 years in a hypercompetitive, globally integrated marketplace.
CNDN graduates have collaborated with and worked for over 1,000 Canadian firms from start-ups and scaleups to industry giants in sectors where Canadian innovation is most needed, such has automotive, aerospace, energy and the environment, healthcare, and defence and security.
Major Research Facilities (MRF’s) in Canada have a big impact on the training of highly qualified people (HQP), and especially on students and post-docs who use these facilities as part of their research. MRFs offer valuable hands-on experiences, powerful virtual capabilities, and meaningful engagements in ‘Big Science’ that are inspiring to young people at formative stages of the educational careers. In contrast, young people whose graduate careers are spent primarily in a single principal investigator’s lab or use only the resources typically available to a single principal investigator will miss out on these experiences or capabilities.
Differences in the subsequent academic and career paths between students who use MRFs and those who don’t are now measurable using online resources such LinkedIn, which now offers profiles of about 25% of the estimated 3 billion people working in 70 million companies around the world.
And these differences can now be measured without painstaking manual labor to look up and copy information about each alumnus or about each employer.
Successful studies: proof of principle
Example result: Alumni from a MRF enjoyed faster career progression. They are about 100% more likely to be in executive and other management positions within 10 years, and about 25% more likely even after 20 years.
Two recent studies by Strategy Policy Economics and TVB Associates on the alumni of two MRFs found impacts such as:
University alumni who used the online services of an MRF during their degree programs enjoyed faster career progression and longer job tenure than those in a control group of alumni from the same research fields, and these correlations last for 20 years.
The hands-on experiences at an MRF inspired undergraduate and Master’s students to achieve greater academic heights, often PhDs. These PhDs then went on to R&D careers in industry in larger proportions than other natural science PhDs (65% vs 51%). Further, most were working in the industry sectors that contribute most directly to innovation in Canada. Alumni who were interviewed for the study attributed part of their career success to the experience using the MRF.
Service offered by TVB Associates
These studies are now offered through TVB Associates as a service. We will conduct, or assist you with conducting, studies of alumni that used a specific MRF or of alumni from research fields that rely heavily on multiple MRFs such as astronomy or experimental particle physics.
TVB Associates can automate the data collection and key aspects of preparing the data for analysis. We have a database of many employers of HQP categorized using the North American Industry Classification System (NAICS) codes as defined by Statistics Canada, which reduces manual labor. We have methods to create valid control groups to ensure meaningful conclusions can be drawn.
These factors allow us to conduct the study on your behalf efficiently without breaking the bank—a much more accessible solution than traditional, manual approaches.
Examples of aggregate insights
These studies can provide a range of insights about the alumni in aggregate:
Analyzing their academic careers reveals:
degrees pursued by alumni after (and potentially inspired by) their interaction with the MRF,
demographic characterization of the alumni according to geographic representation of their institutions and research disciplines, and
differences in the trends in degrees pursued by research fields.
Analyzing their employment career (current employer, first job, or entire employment history) reveals:
the distribution of the alumni across economic sectors,
the distribution of alumni by country (a potential indicator of attraction and retention of HQP),
career success over time as indicated by progression to supervisory, management, or executive positions,
how long alumni stay in their jobs (an indicator of job satisfaction and the fit between the employer and their skills), and
differences in the above in comparison to the control group.
Analyzing both the academic and employment careers together reveals:
how the academic degree or research discipline impacts where the alumni work or their career progression,
whether going on to an advanced degree that doesn’t require use of the MRF changes the career path.
Establishing causation through interviews
The insights gleaned from aggregate analysis may show correlations of interest that can be explored further through interviews. The data obtained in the study will identify the alumni that exhibit the trend of interest and provide a means to contact them for an interview (i.e. through LinkedIn). As an option, we can interview a sample of alumni on your behalf, which may result in testimonials or qualitative data that provides evidence for a causal interpretation of the correlations.
Tailoring the study
TVB Associates will tailor the study to your specific interests, and if applicable, correlate the data with other information that you may have available or that may be obtained elsewhere. For example, if you have a records of industry collaborators, we can determine the extent to which these collaborators hire the alumni. Or if employment of alumni in academic research fields is of interest, we can collect their records in online awards databases (e.g. as published by NSERC and CFI) and analyze it for further insights.
What is required for a study?
A minimum requirement is records of former students and post-docs containing names, university affiliation, and a year associated with their engagement in the relevant research. Additional fields, such as research disciplines (e.g. physics, chemistry, engineering) may be valuable as well (e.g. to establish a valid control group). While a spreadsheet format is ideal, we may be able to convert the data from another format (e.g. if only hard-copy records are available from 20 years ago).
The larger the dataset of alumni the better for statistical reliability and to enable some of the best analysis that involves dividing the alumni into various categories. Based on our previous studies, we can expect to positively identify at least 25% of the alumni using automated methods.
More information
Please contact us to discuss your interests in measuring impacts on training of highly qualified people, and we will explore how we can help.
Demonstrating socioeconomic impact from the training HQP in research has always been difficult, because of challenges associated with tracking graduates and following their subsequent educational and professional careers over time.
The emergence of career-oriented social networking, however, has provided valuable tools that can be used for this purpose. The value of any social network depends greatly on its number of users. The biggest career-oriented network, LinkedIn, has seen a surge in usage since Microsoft took it over in 2016, and its user base is now over 600 million — about 20% of the estimated 3 billion people working in 70 million companies around the world.
Many HQP provide their career information on LinkedIn
The beauty of LinkedIn for tracking HQP is that individuals openly volunteer career information that would otherwise be confidential and very difficult to get. A high proportion of HQP in North America have LinkedIn profiles or can be otherwise identified online, and this includes professionals of all ages. For example, in 2018, I performed internet searches for former students and post-docs who used the now-closed Canadian Neutron Beam Centre (CNBC) for research as part of their graduate or undergraduate programs at Canadian universities, going back as far 1984. I found 75% of these alumni online, and nearly 60% on LinkedIn. Furthermore, 44% of LinkedIn users are women, which is similar to the proportion of women in the workforce overall, suggesting there is little gender-bias in the data, at least at a very high level (however, there can be difficulty in identifying individuals, often women, who have changed their surname).
LinkedIn data reveals where alumni are working now
Some of the simplest results to obtain are the institutions where alumni are working now. For the study for the CNBC, for example, showed that almost 80% of the alumni were working in the sectors that contribute most directly to Canadian innovation: manufacturing, higher education, and professional and technical services. Furthermore, a higher proportion of CNBC alumni with PhDs were working in industry (65%) over academia, as compared to the average for natural sciences PhDs in Canada (51% half stay in academia, according to StatsCan data).
LinkedIn data reveals individual educational and professional paths
LinkedIn data is especially useful for observing alumni’s educational and professional paths over time, because most users treat their profiles like an online resume, listing their record of degrees and professional positions. Such longitudinal data was essential to obtaining valuable insights in the study for the CNBC, such as:
Participation in research at the CNBC as an undergraduate student was a strong predictor of earning a graduate degree: Of the undergraduate students who came to the CNBC for a research project, 60% went on to achieve a graduate degree. In fact, most of these alumni went beyond a single Master’s degree: 40% of the undergraduate students later achieved a PhD, and another 14% earned two Master’s degrees. These rates of academic achievement are far higher than is typical for Canada as a whole: According to StatsCan data, only 44% of all undergraduate students in Canada who are surveyed upon graduation stated intention to pursue further education of any kind. The percentage of students who attain higher degrees is, of course, much lower than those who intended to do so.
Figure 1 Highest degree attained by undergraduate and Master’s students who came to the CNBC for a research project (Data source: LinkedIn). “All Canada” data is the fraction of recent graduates who say they intend to pursue further education of any kind (StatsCan).
Participants in research at the CNBC have enjoyed subsequent career progression: The LinkedIn data showed that alumni with greater years of experience tended to fill more senior positions, while more recently graduated alumni have a greater share of non-supervisory positions.
Figure 2 The level of seniority of the most recent employment positions attained by CNBC student alumni as a function of the number of years that have passed since they attended the CNBC (Data source: LinkedIn).
In the case of the CNBC study, a sample of alumni were contacted via LinkedIn and interviewed. Alumni interviewed attributed their experience of doing research at the CNBC with motivating them to pursue research and development or related technical careers in industry and with helping them develop skills that have helped them in their careers. While not scientifically conclusive, the interview results provide evidence for interpreting some causation in the above observations.
Why aren’t more institutions using LinkedIn data to demonstrate impact from training HQP?
Despite the potential that LinkedIn data holds, I have seen few studies that seek to use the data to full advantage. A notable exception is the 10,000 PhDs project, in which the University of Toronto made a significant investment of effort to identify 88% of its PhD graduates from 2000-2015 online. Many of these alumni were found on LinkedIn. The U of T study analyzed their first and current employment statuses. That study provided valuable insights into employment prospects after earning a PhD, and how that employment differs across fields of study.
Perhaps LinkedIn data has not been used to its full potential because several years ago, its utility for such studies was not as great due to lower usage levels, and one could have reasonably questioned its long-term viability as a platform. But the activity on LinkedIn has greatly increased in recent years and it must now be taken seriously.
Other issues could relate to interpreting the data, data privacy, or the labor required to gather and analyze the data. These issues are discussed next.
Benchmarking to aid data interpretation
A typical challenge in demonstrating impact from training HQP is a lack of reference points to know if the results are excellent or below average. If one believes the results will be ambiguous, then there is less motivation to pursue the analysis.
The key to resolving the ambiguity is to determine appropriate benchmarks and build them into the study. Sometimes the data can be compared with insights from other sources, such as StatsCan as I have done in some of the above examples, to assist with the interpretation. Another option is to conduct the same analyses on random samples of comparator groups (e.g. students who were not involved in research, or were from other institutions distributed across Canada). A comparator group would be useful to interpret the above data on CNBC alumni career progression, for example.
Data privacy
Although LinkedIn users volunteer their career information online, there are still data privacy issues to be considered in collecting and storing the information. LinkedIn users retain the right to remove their data from the site. Systematic duplication of their data by third parties increases the possibility of leakage, which in turn undermines their control over their data.
The U of T study reported that student researchers who conducted the online searches were trained on confidentiality. They entered the data they found into secure servers, at which point they no longer had access to the data. None of the data was stored on personal computers at any time.
With reasonable precautions such as these, data privacy issues need not be a barrier to using the LinkedIn data.
Labor to gather and analyse the data
The labor to gather and analyze the data is perhaps the biggest barrier to using LinkedIn data to its full potential. Few professionals at institutions have time to find large samples of alumni and manually input data from websites into a spreadsheet or database. Longitudinal analyses and benchmarking multiply the amount of data to be found and processed.
The U of T study overcame the labor barrier by using inexpensive part-time student researchers. It reported a $50,000 budget for a team to do the searches and data entry over an 8-month period. The value of staff time to perform subsequent analysis on the data and publish the results can be assumed to be in addition to this budget.
There are also smart ways of automating much of the searching, data entry and data analysis. For the CNBC study, I was fortunate to partner with a consulting firm that had a knack for writing scripts for these purposes. These scripts were key to obtaining results at a reasonable cost.
Furthermore, narrowing the scope of the study to the HQP trained by one or more strategic research facilities at a university can be useful to reduce costs compared to examining the HQP trained by an entire university.
Conclusion and questions for further discussion
The usefulness of LinkedIn as a source of data for demonstrating impact from training HQP has greatly increased in recent years. Research institutions are just beginning realize its potential.
Are you thinking about how to show the value of training HQP in research? What kinds of messages would you like to be able to communicate to governments and research granting agencies, but don’t yet have the evidence to support them? Are you gathering evidence of impact from a major research facility to support its upcoming funding renewal?
Have you been involved in studies using LinkedIn or using general online searches to find and your research alumni? What lessons have you learned? What are your current practices to benchmark the data, respect data privacy, or manage costs?
