Pre-trained Transformers are now ubiquitous in natural language processing, but despite their high end-task performance, little is known empirically about whether they are calibrated. Specifically, do these models' posterior probabilities provide an accurate empirical measure of how likely the model is to be correct on a given example? We focus on BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) in this work, and analyze their calibration across three tasks: natural language inference, paraphrase detection, and commonsense reasoning. For each task, we consider in-domain as well as challenging outof-domain settings, where models face more examples they should be uncertain about. We show that: (1) when used out-of-the-box, pretrained models are calibrated in-domain, and compared to baselines, their calibration error out-of-domain can be as much as 3.5× lower;(2) temperature scaling is effective at further reducing calibration error in-domain, and using label smoothing to deliberately increase empirical uncertainty helps calibrate posteriors out-of-domain. 1

BackgroundThe coverage of community-based maternal, neonatal, and child health (MNCH) services remains low, especially in hard-to-reach areas. We evaluated the effectiveness of a mobile-phone–and web-based application, Innovative Mobile-phone Technology for Community Health Operations (ImTeCHO), as a job aid to the government’s Accredited Social Health Activists (ASHAs) and Primary Health Center (PHC) staff to improve coverage of MNCH services in rural tribal communities of Gujarat, India.Methods and findingsThis open cluster-randomized trial was conducted in 22 PHCs in six tribal blocks of Bharuch and Narmada districts in India. The ImTeCHO mobile-phone–and web-based application included various technology-based job aids to facilitate scheduling of home visits, screening for complications, counseling during home visits, and supportive supervision by PHC staff. Primary outcome indicators were a composite index calculated based on coverage of important MNCH services and coverage of at least two home visitations by ASHA within the first week of birth. Primary analysis was intention to treat (ITT). Generalized Estimating Equation (GEE) was used to account for clustering. Eleven PHCs each were randomly allocated to the intervention (280 ASHAs, population: 234,134) and control (281 ASHAs, population: 242,809) arms. The intervention was implemented from February, 2016 to January, 2017. At the end of the implementation, 6,493 mothers were surveyed. Most of the surveyed women were tribal (5,571, 85.8%), and reported having a government-issued certificate for living below poverty line (4,916, 75.7%). The coverage of at least two home visits within first week of birth was 32.4% in the intervention clusters compared to 22.9% in the control clusters (adjusted effect size 10.2 [95% CI: 6.4, 14.0], p < 0.001). Mean number of home visits within first week of birth was 1.11 and 0.80 for intervention and control clusters, respectively (adjusted effect size 0.34 [95% CI: 0.23, 0.45], p < 0.001). The composite coverage index was 43.0% in the intervention clusters compared to 38.5% (adjusted effect size 4.9 [95% CI: 0.2, 9.5], p = 0.03) in the control clusters. There were substantial improvements in coverage home visits by ASHAs during antenatal period (adjusted effect size 15.7 [95% CI: 11.0, 20.4], p < 0.001), postnatal period (adjusted effect size 6.4, [95% CI: 3.2, 9.6], p <0.001), early initiation of breastfeeding (adjusted effect size 7.8 [95% CI: 4.2, 11.4], p < 0.001), and exclusive breastfeeding (adjusted effect size 13.4 [95% CI: 8.9, 17.9], p < 0.001). Number of infant and neonatal deaths was similar in the two arms in the ITT analysis. The limitations of the study include potential risk of inaccuracies in reporting events that occurred during pregnancy by the mothers and the duration of intervention being 12 months, which might be considered short.ConclusionsIn this study, we found that use of ImTeCHO mobile- and web-based application as a job aid by government ASHAs and PHC staff improved coverage and quality of MNCH ser...

The Government of India has started a new scheme aimed at offering sanitary pads at a subsidized rate to adolescent girls in rural areas. This paper addresses menstrual health and hygiene practices among adolescent girls in a rural, tribal region of South Gujarat, India, and their experiences using old cloths, a new soft cloth (falalin) and sanitary pads. Qualitative and quantitative data were collected in a community-based study over six months, with a pre-and post- design, among 164 adolescent girls from eight villages. Questions covered knowledge of menstruation, menstrual practices, quality of life, experience and satisfaction with the different cloths/pads and symptoms of reproductive tract infections. Knowledge regarding changes of puberty, source of menstrual blood and route of urine and menstrual flow was low. At baseline 90% of girls were using old cloths. At the end of the study, 68% of adolescent girls said their first choice was falalin cloths, while 32% said it was sanitary pads. None of them preferred old cloths. The introduction of falalin cloths improved quality of life significantly (p<0.000) and to a lesser extent also sanitary pads. No significant reduction was observed in self-reported symptoms of reproductive tract infections. Falalin cloths were culturally more acceptable as they were readily available, easy to use and cheaper than sanitary pads.

BackgroundA new cadre of village-based frontline health workers, called Accredited Social Health Activists (ASHAs), was created in India. However, coverage of selected community-based maternal, newborn and child health (MNCH) services remains low.ObjectiveThis article describes the process of development and formative evaluation of a complex mHealth intervention (ImTeCHO) to increase the coverage of proven MNCH services in rural India by improving the performance of ASHAs.DesignThe Medical Research Council (MRC) framework for developing complex interventions was used. Gaps were identified in the usual care provided by ASHAs, based on a literature search, and SEWA Rural's1 three decades of grassroots experience. The components of the intervention (mHealth strategies) were designed to overcome the gaps in care. The intervention, in the form of the ImTeCHO mobile phone and web application, along with the delivery model, was developed to incorporate these mHealth strategies. The intervention was piloted through 45 ASHAs among 45 villages in Gujarat (population: 45,000) over 7 months in 2013 to assess the acceptability, feasibility, and usefulness of the intervention and to identify barriers to its delivery.ResultsInadequate supervision and support to ASHAs were noted as a gap in usual care, resulting in low coverage of selected MNCH services and care received by complicated cases. Therefore, the ImTeCHO application was developed to integrate mHealth strategies in the form of job aid to ASHAs to assist with scheduling, behavior change communication, diagnosis, and patient management, along with supervision and support of ASHAs. During the pilot, the intervention and its delivery were found to be largely acceptable, feasible, and useful. A few changes were made to the intervention and its delivery, including 1) a new helpline for ASHAs, 2) further simplification of processes within the ImTeCHO incentive management system and 3) additional web-based features for enhancing value and supervision of Primary Health Center (PHC) staff.ConclusionsThe effectiveness of the improved ImTeCHO intervention will be now tested through a cluster randomized trial.

BackgroundEven though the caesarean section is an essential component of comprehensive obstetric and newborn care for reducing maternal and neonatal mortality, there is a lack of data regarding caesarean section rates, its determinants and health outcomes among tribal communities in India.ObjectiveThe aim of this study is to estimate and compare rates, determinants, indications and outcomes of caesarean section. The article provides an assessment on how the inequitable utilization can be addressed in a community-based hospital in tribal areas of Gujarat, India.MethodProspectively collected data of deliveries (N = 19923) from April 2010 to March 2016 in Kasturba Maternity Hospital was used. The odds ratio of caesarean section was estimated for tribal and non-tribal women. Decomposition analysis was done to decompose the differences in the caesarean section rates between tribal and non-tribal women.ResultsThe caesarean section rate was significantly lower among tribal compared to the non-tribal women (9.4% vs 15.6%, p-value < 0.01) respectively. The 60% of the differences in the rates of caesarean section between tribal and non-tribal women were unexplained. Within the explained variation, the previous caesarean accounted for 96% (p-value < 0.01) of the variation. Age of the mother, parity, previous caesarean and distance from the hospital were some of the important determinants of caesarean section rates. The most common indications of caesarean section were foetal distress (31.2%), previous caesarean section (23.9%), breech (16%) and prolonged labour (11.2%). There was no difference in case fatality rate (1.3% vs 1.4%, p-value = 0.90) and incidence of birth asphyxia (0.3% vs 0.6%, p-value = 0.26) comparing the tribal and non-tribal women.ConclusionSimilar to the prior evidences, we found higher caesarean rates among non-tribal compare to tribal women. However, the adverse outcomes were similar between tribal and non-tribal women for caesarean section deliveries.