Challenges of Large-Scale Data Processing in the 1990s: The IPUMS Experience

Magnuson, Diana L.; Ruggles, Steven

doi:10.1109/mahc.2022.3214736

Cited by 3 publications

(5 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The Institute for Social Research and Data Innovation and its flagship data project, IPUMS, has its roots in the 1880 Historical Census Project, a Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD) funded project to create a 1-in-100 public use microdata sample of the 1880 U.S. census of population. Housed in the History Department at the University of Minnesota, historical demographers and co-principal investigators Steven Ruggles and Russell Menard conceived of extending back the series of public-use microdata samples already in existence (1900,1910,1940,1950,1960,1970) (Magnuson and Ruggles, 2022). Once the completed 1880 PUMS was disseminated, researcher feedback was overwhelmingly enthusiastic.…”

Section: Institutional Historymentioning

confidence: 99%

“…Once the completed 1880 PUMS was disseminated, researcher feedback was overwhelmingly enthusiastic. 4 Funding to complete the decennial population series-- 1850-1870 and 1920-1930 and updates to 1900 and 1910--would come to the University of Minnesota between 1992 and 2002 (Magnuson and Ruggles, 2022). 5 By 1991 ten machine readable public use microdata samples covering the decennial censuses of population from 1880 and 1990 were publicly available or under development (for 1850, 1880, 1900, 1910, 1940, 1950, 1960, 1970, 1980, and 1990).…”

Section: Institutional Historymentioning

confidence: 99%

“…Between 1985 and 1991, Steven Ruggles "developed a set of FORTRAN programs that recoded selected variables into a common format across the available census samples, created subsets of the samples that were of manageable size, and pooled multiple censuses into a single file" (Magnuson and Ruggles, 2022). Initially Ruggles used a "lowest common denominator" approach for variable codes when combining samples, which naturally resulted in significant loss of information.…”

Section: Institutional Historymentioning

confidence: 99%

“…In 1993, IPUMS data were disseminated through an anonymous file transfer protocol (FTP) site and two years later the IPUMS website launched its own data extract system. Hypertext variable-level documentation became available in 1997 (Magnuson and Ruggles, 2022).…”

Section: Institutional Historymentioning

confidence: 99%

See 3 more Smart Citations

Expanding our perspective: Building a sustainable metadata culture

Magnuson

Thomas

2023

Self Cite

View full text Add to dashboard Cite

The Institute for Social Research and Data Innovation (ISRDI) at the University of Minnesota submitted an application for approval to the Core Trust Seal (CTS) in June 2022. In the course of the protracted process to prepare ISRDI materials for the application, we learned five lessons that expanded our perspective on the role of the archive within our organization and committed our institute to building a sustainable metadata culture. By taking the time to review the specialized nature of ISRDI as it developed over time, clarify and document the processes that developed as the intitute matured and expanded, and applying the standards and guidelines supported by the CTS, ISDRI is now in a position to more easily identify areas of future process development and address outstanding needs in terms of our documentation and preservation of our work. These lessons are applicable to research organizations that include the responsibility to preserve a record of their work in the mid- and long-term.

show abstract

Section: Institutional Historymentioning

confidence: 99%

Section: Institutional Historymentioning

confidence: 99%

Section: Institutional Historymentioning

confidence: 99%

Section: Institutional Historymentioning

confidence: 99%

See 2 more Smart Citations

Expanding our perspective: Building a sustainable metadata culture

Magnuson

Thomas

2023

Self Cite

View full text Add to dashboard Cite

show abstract

“…Over the last thirty years, IPUMS at the University of Minnesota has created the world's largest accessible database of census and survey microdata (Magnuson and Ruggles 2022). The primary work of IPUMS is data harmonization-making census and survey data compatible across time and space.…”

Section: Introductionmentioning

confidence: 99%

Stewarding our resources: Building a sustainable IPUMS archival document access system

Magnuson

2024

Self Cite

View full text Add to dashboard Cite

IPUMS at the University of Minnesota has created the world’s largest accessible database of census and survey microdata. The IPUMS suite of products contains nine harmonized data products. The largest of these projects, IPUMS International (IPUMS-I) has supported the curation and preservation of ancillary materials received during data acquisition efforts. Archival staff have preserved thousands of unique pieces of census and survey documentation, creating bibliographic records using an extended Dublin Core profile that supports the use of controlled vocabularies to enhance findability for the project staff and outside users. The goal of this curation work was to create a findable, searchable, and downloadable document access system for our internal use and to support IPUMS researchers. This paper describes our experience constructing a tool that supports exploration and dissemination of these archived materials. During this development, we gained valuable insight about stewarding our resources that are applicable to research organizations responsible for curating, preserving, and disseminating archival materials.

show abstract

The IPUMS Business Process Model: Instituting a workflow mapping strategy to support archival processes

Magnuson

2024

View full text Add to dashboard Cite

The IPUMS Preservation Archive is instituting a workflow mapping strategy to further identify IPUMS process and metadata capture points to expand its holdings in the data archive. Drawing on two business process models, the Generic Statistical Business Process Model (GSBPM) and the Generic Longitudinal Business Process Model (GLBPM), archival staff have created an IPUMS Business Process Model (IPUMS BPM). The IPUMS BPM reflects the use of secondary data sources and the work of harmonization and integration to create a data infrastructure that supports research across time and space. Internally, the IPUMS BPM provides a clear visualization of the IPUMS workflow from external submission of data, harmonization process, documentation, extraction systems, and archival preservation of metadata. The challenge for archival staff is furthering the understanding and adoption of the IPUMS BPM within the IPUMS project groups, and to identify metadata production points that require the intervention of the archive for provenance and preservation purposes. It is part of an on-going effort to clearly define the role of the archive within IPUMS as an integral part of IPUMS organization and workflow. This paper identifies the value of instituting this mapping approach to gain a clearer understanding of the role of the archive within project work cycles, points where production and preservaton activities intersect, and opportunities to expand archival holdings.

show abstract

Challenges of Large-Scale Data Processing in the 1990s: The IPUMS Experience

Cited by 3 publications

References 16 publications

Expanding our perspective: Building a sustainable metadata culture

Expanding our perspective: Building a sustainable metadata culture

Stewarding our resources: Building a sustainable IPUMS archival document access system

The IPUMS Business Process Model: Instituting a workflow mapping strategy to support archival processes

Contact Info

Product

Resources

About