Much focus has been on the interaction of programmed cell death ligand 1 (PD-L1) on malignant B cells with programmed cell death 1 (PD-1) on effector T cells in inhibiting antilymphoma immunity. We sought to establish the contribution of natural killer (NK) cells and inhibitory CD163 monocytes/macrophages in Hodgkin lymphoma (cHL) and diffuse large B-cell lymphoma (DLBCL). Levels of PD-1 on NK cells were elevated in cHL relative to DLBCL. Notably, CD3CD56CD16 NK cells had substantially higher PD-1 expression relative to CD3CD56CD16 cells and were expanded in blood and tissue, more marked in patients with cHL than patients with DLBCL. There was also a raised population of PD-L1-expressing CD163 monocytes that was more marked in patients with cHL compared with patients with DLBCL. The phenotype of NK cells and monocytes reverted back to normal once therapy (ABVD [doxorubicin 25 mg/m, bleomycin 10 000 IU/m, vinblastine 6 mg/m, dacarbazine 375 mg/m, all given days 1 and 15, repeated every 28 days] or R-CHOP [rituximab 375 mg/m, cyclophosphamide 750 mg/m IV, doxorubicin 50 mg/m IV, vincristine 1.4 mg/m (2 mg maximum) IV, prednisone 100 mg/day by mouth days 1-5, pegfilgrastim 6 mg subcutaneously day 4, on a 14-day cycle]) had commenced. Tumor-associated macrophages (TAMs) expressed high levels of PD-L1/PD-L2 within diseased lymph nodes. Consistent with this, CD163/PD-L1/PD-L2 gene expression was also elevated in cHL relative to DLBCL tissues. An in vitro functional model of TAM-like monocytes suppressed activation of PD-1 NK cells, which was reversed by PD-1 blockade. In line with these findings, depletion of circulating monocytes from the blood of pretherapy patients with cHL and patients with DLBCL enhanced CD3CD56CD16 NK-cell activation. We describe a hitherto unrecognized immune evasion strategy mediated via skewing toward an exhausted PD-1-enriched CD3CD56CD16 NK-cell phenotype. In addition to direct inhibition of NK cells by the malignant B cell, suppression of NK cells can occur indirectly by PD-L1/PD-L2-expressing TAMs. The mechanism is more prominent in cHL than DLBCL, which may contribute to the clinical sensitivity of cHL to PD-1 blockade.

The purpose of this technical report is two-fold. First of all, it introduces a suite of challenging continuous control tasks (integrated with OpenAI Gym) based on currently existing robotics hardware. The tasks include pushing, sliding and pick & place with a Fetch robotic arm as well as in-hand object manipulation with a Shadow Dexterous Hand. All tasks have sparse binary rewards and follow a Multi-Goal Reinforcement Learning (RL) framework in which an agent is told what to do using an additional input. The second part of the paper presents a set of concrete research ideas for improving RL algorithms, most of which are related to Multi-Goal RL and Hindsight Experience Replay.

Dealing with sparse rewards is one of the biggest challenges in Reinforcement Learning (RL). We present a novel technique called Hindsight Experience Replay which allows sample-efficient learning from rewards which are sparse and binary and therefore avoid the need for complicated reward engineering. It can be combined with an arbitrary off-policy RL algorithm and may be seen as a form of implicit curriculum. We demonstrate our approach on the task of manipulating objects with a robotic arm. In particular, we run experiments on three different tasks: pushing, sliding, and pick-and-place, in each case using only binary rewards indicating whether or not the task is completed. Our ablation studies show that Hindsight Experience Replay is a crucial ingredient which makes training possible in these challenging environments. We show that our policies trained on a physics simulation can be deployed on a physical robot and successfully complete the task. The video presenting our experiments is available at https://goo.gl/SMrQnI.