Demography for program evaluation and policy
At its broadest, applied demography in the field of program evaluation and policy includes two halves: the collection and standardization of multi-agency data and documentation, as well as the interpretation of this information to answer questions about the effectiveness of programs and policies. These two halves present themselves as a Venn diagram: an applied demographer may work entirely in one area, entirely in the other, or commonly in both. These roles fulfill a variety of demands. It is widely unacceptable for offices collecting their own data to make up their own bespoke demographic labelling schemes, but they will require someone with broader inter-agency perspective to advise on the best practices. Geographers, ethnographers, criminologists, and so forth may lack the overall views of an applied demographer needed to ensure their data works well on its own and together. Also, offices should no longer assume they will keep their own silos of data, with an evaluator forced to assume the role of a needy supplicant seeking out whatever information they can get — instead, there are increasingly cooperative and even centralized approaches to sharing information. Lastly, conflicts of interest and professional specializations make it challenging for offices to report on their own successes or failures in valid and clear manners.
Some applied demographers focus on the curated acquisition of information from many different areas of government, to provide centralized resources such as an education statistics office. This involves receiving regular updates from various agencies, as well as determining best practices to align with industry standards, allow longitudinal tracking, build cohesive interpolatory data labelling systems, and protect privacy. Their individual role may involve little interpretation, but still require the application of demographic principles to ensure the information they release can fulfill a purpose. Other applied demographers in program evaluation and policy focus instead on using agency information to provide rapid and actionable reports centered around questions of efficacy, optimization, and continuation of funding. They may work in an area of government with little or no centralized data resource, but still have to find sources to fulfill their task.
As mentioned above, many applied demographers in program evaluation and public policy work in both of these areas. In one broadest and most scaled-up roles for an applied demographer in program evaluation and policy, a practitioner will oversee a statewide office that collects information from a wide range of agencies, and aside from providing this as a public resource they will also use this to publish reports based on this information as required by law or as commissioned by individual offices. Another example of a robust role would be a practitioner working at the federal or inter-governmental level who is involved with setting standards for reporting so that racial and other demographic characteristics can be interpolated, grouped, and compared, and who liaises with querants at both the acquisition and reporting stages.
There is a “third half” to the role of the applied demographer in this field: that of dissemination. Aside from the integrity that comes with evaluation being assigned to a separate department than the one overseeing a particular program or policy implementation, the demands of dissemination require specialists in themselves. On the one hand, this can be very direct: a policymaker can contact an office that presides over data and interpretation to prepare a report on an area of interest. In this case, the commissioning and dissemination is publicly available but otherwise between two teams. Although practitioners are generally proscribed in regards to advocating for particular policies or candidates, this impartiality is part of the value they bring: they can provide the information needed for action no matter how controversial, without themselves being activists on that issue, and according to legal frameworks (e.g. federal mandates) and pragmatic considerations (e.g. funding). However, dissemination can also be extremely broad. In the widest examples commonly found, a statewide statistics office may publish county-level reports that are then used by counties to guide their decisions for the future. For example, the state may pass education legislation that focuses on specific measurements of achievement. Later, a statewide office may report on how districts and/or counties are performing longitudinally on these measurements. Importantly, the demographic perspective means taking a look at issues between various racial, gender, and socioeconomic groupings as standard practice. This means that not only can local administrators see how they are performing, but they can also get very granular information to assess where they need to focus the most.
Dissemination may be considered a speciality in itself for the applied demography practitioner. Reports must be readable, using techniques and results that are clearly communicated. In order to be actionable, it requires an understanding of the primary readerships for the report and their work. Aside from these issues which impact the construction of the report and its language, the dissemination must also be timely. In many cases, this means timeframes of weeks and months and opposed to years and decades. To achieve this, the practitioner should achieve the level of precision necessary to have certainty that data, forecasts, and results would not undergo any major reversal upon further study, along with achieving a level of depth necessary for the report to actually be meaningful. In addition, reports, communiques, and other publications should be properly packaged for digital-, print-, and presentation-oriented transmission, including visuals that ensure information is not only communicated at one degree of separation from the practitioner themselves (e.g. from the report to a primary reader) but also that that readers can in turn extract salient components and communicate them to their own audiences as well (e.g. a policymaker making a speech).
Different practitioners bring their own personality and career path to these broad observations on applied demography in the area of program evaluation and policy. Dr. James Sanders, Director of Strategy at University of Utah Health, described the demographic framework as both a frame of thought and a very real-life paradigm. The demographic framework involves tools to make population forecasts — e.g. the size of a population overall, as well as by gender, age, and so forth. Dr. Sanders gave an example of how the demographic framework allowed his team to predict the directions in which healthcare services should expand in order to best serve a growing community, based on considerations like the changing age structure of the population and other factors such as immigration. Importantly, Dr. Sanders also pointed out that this is more than just a thought exercise: the demographic framework helps to describe very real pathways for inequalities and disparities. Health care is surely just one system which shows selectivity according to demography such as age, gender, and other fundamental demographic characteristics; this type of pattern is also very relevant for education, housing, and other systems where the government interacts with society to make overall improvements and correct for biases.
Dr. Beth Gifford has a multi-hyphenate career with multiple professional and professorial roles, including as Director at both Durham Children's Data Center and Duke University’s Program Evaluation Services. While Dr. Sanders explained the demographic framework as essentially the study of populations with dimensions of demographic descriptiveness, Dr. Gifford described a range of tools. This is a common pattern among many applied demography practitioners: the hyperopia of generalizing the demographic framework as being based on principles and groupings; and the myopia of particularizing the demographic framework as being based on mathematical and analytical tools. In both cases, practitioners are describing essentially the same demographic framework but in different ways. In Dr. Gifford’s case, her more particular approach stemmed from her elevated professional role and the deduplication she conducts in identifying the same ways and different ways that a wide range of different professionals under her are approaching the same problems. In her case, she described the demographic framework from the perspective of what skills can be brought onto a team that might also include sociologists, pediatricians, and epidemiologists, among others. This shares a common origin with Dr. Sanders’ description — a bifurcation of using the same demographic framework, not a converge from two totally different demographic frameworks.
Dr. Vivien Chen is the Senior Research Analyst for the state of Washington Education and Research Data Center, and also described the demographic framework in her work. Notably, she described forming a demographic framework in her work — not just applying a demographic framework. This suggested a sort of genesis, and she further described how the demographic framework is the way she develops research approaches that incorporates processes (fertility, mortality, migration, and immigration) as well as characteristics (wage, immigration status, etc), and associates these demographic processes and characteristics with policies. This reflects her role in analysis and developing research approaches, and being confronted with policy questions that require formation of a research framework that incorporates demography.
With all these perspectives on the demographic framework, there is an interplay between applied demography and academic demography. All three professionals had very formal doctorate backgrounds, yet have found themselves thrust into scenarios with sometimes problematic tensions between ideals of academic demography and the exigencies of their work. This tension is where applied demography has arisen as its own field meriting its own theorization and training. When asked to describe applied versus academic demography, these three practitioners voiced different tensions according to the given wavelengths of their work.
Dr. Sanders emphasized the importance of speed in applied demography. An academic report that reached a statistical confidence of 75% when just a few more weeks or months could have achieved 95% confidence would be rightly ridiculed. However, in applied demography weeks or months may flatly be unavailable or may mean the report is already outdated. For example, a quarterly tracker would be useless if always published six months late rather than within a few weeks. An academic demographer might resolve this by looking retrospectively at several years in a report so that the lateness is less visible against the backdrop of two years, five years, or even decades. However, an applied demographer will need to meet shorter deadlines and know with certainty that despite weaker statistical precision there is no substantial risk that their findings would totally change with further research. Dr. Sanders described how an applied demographer may obtain 80% precision in one time-frame but it could take double or more that amount of time to reach 95% precision. The academic demographer must provide near-absolute certainty, in addition to their research being reproducible. However, this component of an academic’s work may just refine the main finding (the substance of the work, and what would be of interest to an applied demographer’s audience) and add a level of rigor and airtightness that is primarily of interest to other academics and only in the academic field itself (and not be of substantial interest to an applied demographer’s audience). This has trade-offs, clearly, and Dr. Sanders described that honest conversations and caveats are required about research that may be published in weeks as opposed to months. However, this may be the only approach possible in a scenario where incompleteness may have very real benefit as the only way to get results ahead of months and years where the environment may change and an academic report at that point would have no practical benefit whatsoever.
However, the difference is not only in what is published, but in how it is published. Harkening to her high-level supervisory role, Dr. Giffords emphasized that academic and applied demographers have similar backgrounds and use mostly the same tools, but have different stresses and teams involved in their work. While an academic demographer may self-fund noncommissioned research using grants they apply to on their own, an applied demographer will often work within a team that is funded statically and/or by commission. Dr. Chen tied together Dr. Sanders’ and Dr. Gifford’s perspectives by describing the many facets of her own acclimatization from an academic environment to an applied environment. Early in her career path, she struggled for several years with what research methods to use, and how to navigate politics, government systems, and state agencies. Her tendency was toward more sophisticated research methods, but time, politics, and audience comprehension made her feel restrained. Over time, she adapted to not only doing quality research but also doing quality research that people could understand. This transformation was not just on her shoulders: she also elevated the knowledge of major stakeholders, allowing her to incorporate regression analysis into her center’s reports. While many applied demographers focus on characteristics and rates, methods such as regression analysis, difference-in-difference analysis, and regression discontinuity design are part of an applied demographer’s graduate-level training. While they are not always used in reports, Dr. Chen described how having a strong academic foundation has allowed her to safeguard standards for research quality, and ultimately allowed her to develop consumers’ acumen to a point where she could incorporate more sophisticated techniques.