To start answering these questions, let's look at what an advisor actually does. An advisor should be able to help you plan your coursework in the most effective way to get you where you want to be in terms of academic and career choices. They should be able to help you pick out courses that will give you a leg up in graduate programs and internships.
Beyond coursework, your adviser is probably going to be the faculty member who knows the most about you as a student and a person. They will likely be one of the first people you turn to when asking for a letter of recommendation. (In fact, some internships require a letter from your academic advisor specifically.) Therefore, you want your adviser to be someone with whom you feel comfortable speaking and sharing your goals, past and other pertinent issues.
While your academic adviser is here for your benefit, you have responsibilities as well. While your advisor may be able to suggest some courses and careers based on your interest, no one can choose a path for you. So, it's important to make your choices about what you want out of your experience at St. Olaf and to allow your advisor to help you achieve that. Likewise, you need to be willing to give your advisor the information they need to write letters of recommendation. Therefore, you want your advisor to be someone you trust and with whom you feel comfortable.
And sometimes that just doesn't work out. Many people change majors or ambitions over their course of study at St. Olaf. Since advisors are assigned on initial major interests, and most faculty members don't know the requirements outside of their department very well (if at all), a dramatic shift in the major course of study will likely necessitate a change of advisors. Even if your advisor is still able to help you graduate, you might consider shifting advisors to someone with interests closer to your own. It's no biggie, really. All you have to do is find a new advisor with an area of expertise closer to your own, fill out a form and obtain a couple signatures and you're home free. People change their career paths all the time; faculty members are used to that. Still, your advisor is the person who knows you best academically; you might want to keep the relationship alive for letters of recommendation.
But what about when you feel like your advisor has been neglectful, has given you bad advice or you just don't like to be around them? Here the situation can get a little more hairy.
Oftentimes an absentee advisor will be aware that they've been a little hard to contact. In the case of a sabbatical, advisors may even prompt a student to switch in the mean time. In any event, an advisor is here to help you get to where you need to be. If your needs aren't being met, try to talk to your advisor about the problem. If they refuse to work with you more, then there's no need to waste your time and theirs attempting to revive a failing relationship.
Still, there's no need to burn bridges. Approach the subject tactfully, explain why it is that your needs aren't being met and ask them to sign the advisor changing form. The important thing is to remain cool -- after all, a student/advisor relationship is largely for your edification and you're entitled to have a positive experience.
If you really feel like you've been wronged (keep in mind that taking your advisor's advice doesn't guarantee that you'll graduate in four years, that you'll get into your classes or that graduate schools will flock to you), have a chat with your department chair. If you've made the effort to be prepared for your meetings with them, kept them informed on your academic progress and provided for their needs, they owe you the same. Letting your department chair know about a legitimately bad experience may spare others the same.
At the end of the day, a relationship with an advisor is a two-way street: You prepare for your meetings and supply them with the information needed to help you, and they guide you on your academic journey. But when its time for the affair to end, just make sure remain cool, explain your reasons and keep the conversation cordial.