Pembrolizumab in Treating Patients With Hormone Receptor Positive, Localized Inflammatory Breast Cancer Who Are Receiving Hormone Therapy and Did Not Achieve a Pathological Complete Response to Chemotherapy

This phase II trial studies how well pembrolizumab works in treating patients with hormone receptor positive inflammatory breast cancer that has not spread to other parts of the body, who are receiving hormone therapy and did not achieve a pathological complete response to chemotherapy. Immunotherapy with monoclonal antibodies, such as pembrolizumab, may help the body's immune system attack the cancer, and may interfere with the ability of tumor cells to grow and spread.

Inclusion Criteria: 1. Is willing and able to provide written informed consent for the trial. 2. Is a female or male and \>/= 18 years of age 3. Has histological confirmation of breast carcinoma. 4. Has confirmed inflammatory breast cancer by using international consensus criteria: * Onset: Rapid onset of breast erythema, edema and/or peau d'orange, and/or warm breast, with/without an underlying breast mass * Duration: History of such findings no more than 6 months * Extent: Erythema occupying at least 1/3 of whole breast * Pathology: Pathologic confirmation of invasive carcinoma 5. Did not achieve pathological complete response (pCR) to any chemotherapy that was given with the intention to induce best response prior surgery. pCR is defined as the current American Joint Committee on Cancer (AJCC) breast cancer staging. 6. Is HER2 normal, defined as HER2 0 or 1+ by IHC and negative by FISH if performed; or HER2 is 2+ by IHC and negative by FISH; or HER2 negative by FISH if IHC is not performed. 7. Has positive ER or PR status. ER or PR \>/= 10% 8. Has a performance status of 0-1 on the ECOG Performance Scale. 9. Has adequate organ function as determined by the following laboratory values: ANC \>/= 1,500 /mcL, Platelets \>/=100,000 /mcL, Hgb \>/= 9 g/dL, creatinine levels \< 1.5 x ULN, Total bilirubin \