Efficacy and Safety of Nemtabrutinib (MK-1026) in Participants With Hematologic Malignancies (MK-1026-003)

Inclusion Criteria: * Has an Eastern Cooperative Oncology Group (ECOG) performance status of 0 to 2 within 7 days before C1D1 (the first dose of study treatment) * Has a life expectancy of at least 3 months, based on the investigator assessment * Has the ability to swallow and retain oral medication * Participants who are Hepatitis B surface antigen (HBsAg)-positive are eligible if they have received Hepatitis B virus (HBV) antiviral therapy for at least 4 weeks and have undetectable HBV viral load prior to randomization * Participants with history of Hepatitis C virus (HCV) infection are eligible if HCV viral load is undetectable at screening * Has adequate organ function * Male participants agree to refrain from donating sperm and agree to either remain abstinent from penile-vaginal intercourse as their preferred and usual lifestyle OR agree to use contraception, during the intervention period and for at least the time required to eliminate the study intervention after last dose of study intervention * Female participants assigned female sex at birth who are not pregnant or breastfeeding are eligible to participate if not a participant of childbearing potential (POCBP), or if a POCBP they either use a contraceptive method that is highly effective OR remain abstinent from penile-vaginal intercourse as their preferred and usual lifestyle during the intervention period and for at least to eliminate study intervention after the last dose of study intervention * Participants with Human immunodeficiency virus (HIV) are eligible if they meet all of the following: the CD4 count is \>350 cells/uL at screening, the HIV viral load is below the detectable level, are on a stable ART regimen for at least 4 weeks prior to study entry, and are compliant with their ART Part 1 and Part 2 (Cohorts A to C and J) * Has a confirmed diagnosis of Chronic lymphocytic leukemia/ Small lymphocytic lymphoma (CLL/SLL) with * At least 2 lines of prior therapy (Part 1 only) * Part 2 Cohort A: CLL/SLL participants who are relapsed or refractory to prior therapy with a covalent, irreversible Bruton's tyrosine kinase inhibitor (BTKi), and a B-cell lymphoma 2 inhibitor (BCL2i). CLL participants must have received and failed, been intolerant to, or determined by their treating physician to be a poor phosphoinositide 3-kinase inhibitor (PI3Ki) candidate or ineligible for a PI3Ki per local guidelines * Part 2 Cohort B: CLL/SLL participants who are relapsed or refractory following at least 1 line of prior therapy and are BTKi treatment naive * Part 2 Cohort C: CLL/SLL participants with 17p deletion or tumor protein p53 (TP53) mutation who are relapsed or refractory following at least 1 line of prior therapy * Part 2 Cohort J: CLL/SLL participants whose disease relapsed or was refractory to prior therapy with a covalent/irreversible BTKi and BCL2i. NOTE: As of Protocol Amendment 09, at least 10 CLL/SLL participants whose disease relapsed or was refractory to prior therapy with a covalent/irreversible BTKi, BCL2i and noncovalent/reversible BTKi (all three classes of therapies are required) will be enrolled into Cohort J * Has active disease for CLL/SLL clearly documented to initiate therapy * For SLL participants in Part 2: Has evaluable core or excisional lymph node biopsy for biomarker analysis from an archival or newly obtained biopsy or bone marrow aspirate at Screening (optional for participants enrolling in Part 1) Part 2 (Cohorts D to G) \- Has a confirmed diagnosis of and meets the following prior therapy requirements: * Participants with Richter's transformation who are relapsed or refractory following at least 1 line of prior therapy (Cohort D) * Participants with pathologically confirmed Mantle-cell lymphoma (MCL), documented by either overexpression of cyclin D1 or t (11;14), who are relapsed or are refractory to chemoimmunotherapy and a covalent irreversible BTKi (Cohort E) * Participants with Marginal zone lymphoma (MZL) (including splenic, nodal, and extra nodal MZL) who are relapsed or refractory to at least one prior line of systemic therapy including an anti-CD20-based regimen * Participants with Follicular lymphoma (FL) who are relapsed or refractory to chemoimmunotherapy and immunomodulatory agents (such as lenalidomide based regimen) (Cohort G) * Have measurable disease defined as at least 1 lesion that can be accurately measured in at least 2 dimensions with spiral Computed tomography (CT) scan * Has a lymph node biopsy for biomarker analysis from an archival or newly obtained biopsy or bone marrow aspirate (Cohort D) at Screening Part 2 (Cohort H): confirmed diagnosis of Waldenström's macroglobulinemia (WM); participants who are relapsed or refractory to standard therapies for WM including chemoimmunotherapy and a covalent irreversible BTKi * Has active disease defined as 1 of the following: systemic symptoms, physical findings, laboratory abnormalities, coexisting disease * Has measurable disease, satisfying any of the following: at least 1 lesion that can be accurately measured in at least 2 dimensions with spiral CT scan (minimum measurement must be \>15 mm in the longest diameter or \>10 mm in the short axis); IgM ≥450 mg/dL; or bone marrow infiltration of 10% * Has fresh bone marrow aspirate or a lymph node biopsy for biomarker analysis at Screening or a lymph node biopsy from an archival Exclusion Criteria: * Has active HBV/HCV infection (Part 1 and Part 2) * Has a history of malignancy ≤3 years before providing documented informed consent. Participants with basal cell carcinoma of skin, squamous cell carcinoma of skin, or carcinoma in situ (e.g., breast carcinoma, cervical cancer in situ) that have undergone potential curative therapy are not excluded. Participants with low-risk, early-stage prostate cancer (T1-T2a, Gleason score ≤6, and prostate-specific antigen \<10 ng/mL) either treated with definitive intent or untreated in active surveillance with SD are not excluded * Has active central nervous system (CNS) disease * Has an active infection requiring systemic therapy * Has received prior systemic anti-cancer therapy within 5 half-lives or 4 weeks (if prior therapy was a monoclonal antibody) before C1D1 * Is currently participating in or has participated in a study of an investigational agent or has used an investigational device within 4 weeks prior to the first dose of study intervention * Has any clinically significant gastrointestinal abnormalities that might alter absorption * History of severe bleeding disorders