Personalized Peptide Vaccine in Treating Patients With Advanced Pancreatic Cancer or Colorectal Cancer

This phase I trial studies the side effects and best way to give personalized peptide vaccine in patients with pancreatic or colorectal cancer that has spread to other places in the body and usually cannot be cured or controlled with treatment (advanced). Personalized peptide vaccine is a vaccine developed from patient's own tumor cells and blood in order to use as a biological therapy. Biological therapies, such as personalized peptide vaccine may attack tumor cells and stop them from growing or kill them.

Inclusion Criteria: * Metastatic CRC or PDA planned to or have undergone complete surgical resection (metastatectomy) and also for PDA participants with localized disease planned for primary tumor resection. * Any lines (including zero) of therapy prior to tissue harvest. * Adequate tumor tissue availability * Adults (age ≥ 18) * ECOG PS 0-1 * Life expectancy \>12 months for Cohort C and \>9 months for Cohort D * Adequate organ and marrow function * Ability to understand and the willingness to sign a written informed consent document. * As the effects of a peptide based vaccine, pembrolizumab or APX005M on the developing human fetus are unknown, women of child-bearing potential and men must agree to use adequate contraception at study entry and for the duration of study participation. Should a woman become pregnant or suspect she is pregnant, she should inform her treating physician immediately. Unless surgically sterile by bilateral tubal ligation or vasectomy of partner(s), sexually active participants must use birth control during and for \>120 days after the study. Abstinence is also an acceptable form of birth control. * For Cohort C only: participants must have metastatic CRC and are planned to or have undergone complete metastecomy/ies and agree to have post-operative blood draw for ctDNA testing within 6 weeks following surgical resection. Exclusion Criteria: * History of HIV or AIDS * Patients with brain metastasis * Serious autoimmune conditions * Use of chronic immune suppressive medications. * Uncontrolled concurrent illness including, but not limited to ongoing or active infection, symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia, or psychiatric illness/social situations that would limit compliance with study requirements. * Women of child bearing potential who are pregnant or breastfeeding. Women with a positive pregnancy test at enrollment or prior to administration of vaccine. . * Has history of (non-infectious) pneumonitis that required steroids, evidence of interstitial lung disease or active, non-infectious pneumonitis. * Known history of active TB (Bacillus Tuberculosis). * Hypersensitivity to therapy drugs or their components. * Has a known additional malignancy that is progressing or requires active treatment. * Active coagulopathy. * History of arterial thrombosis within 3 months prior to starting study treatment. * History of New York Heart Association Class 3-4 heart failure or myocardial infarction within 6 months prior to starting therapy. * Known history of Hepatitis B or known active Hepatitis C.