FDIC's advice is to shop around and put the offers to writing.

Once you know what each lender has to offer, negotiate for the best deal that you can. On any given day, lenders and brokers may offer different prices for the same loan terms to different consumers, even if those consumers have the same loan qualifications. The most likely reason for this difference in price is that loan officers and brokers are often allowed to keep some or all of this difference as extra compensation. Generally, the difference between the lowest available price for a loan product and any higher price that the borrower agrees to pay is an overage. When overages occur, they are built into the prices quoted to consumers. They can occur in both fixed and variable-rate loans and can be in the form of points, fees, or the interest rate. Whether quoted to you by a loan officer or a broker, the price of any loan may contain overages.
Have the lender or broker write down all the costs associated with the loan. Then ask if the lender or broker will waive or reduce one or more of its fees or agree to a lower rate or fewer points. You’ll want to make sure that the lender or broker is not agreeing to lower one fee while raising another or to lower the rate while raising points. There’s no harm in asking lenders or brokers if they can give better terms than the original ones they quoted or than those you have found elsewhere.

Once you are satisfied with the terms you have negotiated, you may want to obtain a written lock-in from the lender or broker. The lock-in should include the rate that you have agreed upon, the period the lock-in lasts, and the number of points to be paid. A fee may be charged for locking in the loan rate. This fee may be refundable at closing. Lock-ins can protect you from rate increases while your loan is being processed; if rates fall, however, you could end up with a less favorable rate. Should that happen, try to negotiate a compromise with the lender or broker.